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FOREWORD 

The  Naval  Training  Equipment  Center  has  been  involved  in  a  continuing 
program  of  training  evaluations  for  the  past  several  years.  The  primary 
goal  of  this  effort  is  to  quantify  the  effects  of  the  use  of  training  systems, 
and  with  this  information,  to  improve  present  and  future  training  utilization 
c  id  design  of  training  systems.  A  related  goal  is  to  establish  the  cost 
effectiveness  of  such  systems. 

The  present  study  involved  Device  2F69B,  a  weapon  systems  trainer 
for  me  P-3A  and  P-3B  Aircraft.  This  device  is  designed  to  provide  tactics 
crews  with  team  training  in  the  detection,  tracking,  and  destruction  of 
modern  deep-diving  submarines.  By  carefully  selecting,  varying,  and 
controlling  the  problem  conditions,  the  instructors  should  be  able  to  train 
the  tactics  teams  to  properly  analyze  and  respond  to  situations  likely  to 
occur  on  an  actual  ASW  mission.  The  intent  of  this  study  was  to  test  the 
performance  of  these  teams  to  determine  how  well  the  training  objectives 
were  being  met. 

The  study  results  indicate  that  learning  takes  place  in  the  simulator 
and  that  there  is  positive  transfer  to  the  operational  environment.  There  is, 
however,  room  for  improvement  and  recommendations  are  made  in  several 
areas  for  better  control  of  student  flow  and  for  modification  of  the  training 
curriculum. 

The  importance  of  clarifying  the  capabilities  of  the  trainer  to  the 
Instructors  is  becoming  evident  as  more  training  effectiveness  evaluations 
are  completed.  It  is  appearing  as  a  common  factor,  that  many  training 
devices  are  being  installer  without  adequate,  formal  instruction  for  their 
proper  utilization.  In  cases  when  formal  .nstruction  is  initially  provided, 
it  may  be  lost  through  instructor  turnover.  Thorough  indoctrination  in  the 
proper  use  of  a  training  device  should  be  made  periodically,  in  order  to 
utilize  the  trainer's  rna  imum  capabilities.  A  training  device  can  be  only 
as  good  as  its  managei  v-nt  by  support  and  training  personnel. 

^/JOSEPH  A.  PUIG-' 

Project  Psychologist 


NAVTRAEQUIPCEN  70-C-0258-2 


TABLE  OF  CONTENTS 


6.  otkn.  Page 

i  INTRODUCTION .  i 

Purpose  of  the  Study .  1 

General  Description  of  the  2F69B  Trainer .  1 

Comparison  of  OFT-WST  Segments  for  Evalua¬ 
tion  Purposes  .  2 

II  EXPERIMENT  ONE .  5 

Introduction .  5 

Experimental  Design .  5 

Study  Methods .  5 

Study  Measures .  6 

Crew  Performance  Measures .  6 

Individual  Performance  Measures  ...  7 

Study  One  Results .  9 

Crew  Performance .  9 

Individual  Performance  Measures  ....  12 

III  EXPERIMENT  TWO .  21 

Introduction .  21 

Subject  Population .  21 

Experimental  Design .  21 

Study  Measures .  2  3 

Crew  Performance  Measures .  24 

Individual  Performance  Measures .  24 

Crew  Performance  Measures .  26 

Individual  Performance  Scores .  36 

I"  QUESTIONNAIRE  EVALUATION  OF  DEVICE 

2F69B .  47 

Device  Characteristics  Ratings .  47 

Instructor  Comments .  o2 

CONCLUSIONS  AND  RECOMMENDATIONS .  71 

Conclusions  .  71 

Recommendations .  72 

Appendix  A  .  75 

Appendix  B .  79 

Appendix  C .  95 

Appendix  D  .  Ill 


NAVTRAEQUIPCEN  70-C-0258-2 


LIST  OF  ILLUSTRATIONS 

Figure  Page 

1  Crew:  Navigation  Stabilization  Accuracy .  10 

2  Crew:  Search  Phase  Accuracy .  11 

3  Crew:  Search  Phase  Efficiency  .  . .  11 

4  Crew:  Localization  Phase  Accuracy .  13 

5  Crew:  Localization  Phase  Efficiency .  13 

6  Crew:  Attack  Phase  Accuracy .  14 

7  Crew:  Attack  Phase  Efficiency .  14 

8  Individual:  Navigator  Annotation  Accuracy  and 

Completeness .  15 

9  Individual:  Jezebel  Operator  Annotation  Accuracy 

and  Completeness .  16 

10  Individual:  Julie/ECM  Operator  Annotation 

Accuracy  and  Completeness  (N  =  6) .  17 

11  Individual:  Radar /MAD  Operator  Annotation 

Accuracy  and  Completeness  (N  =  2) .  19 

12  Crew:  Navigation  Stabilization  Procedures .  27 

13  Crew:  Navigation  Stabilization  Accuracy  .  29 

14  Team:  Search  Phase  Procedures .  32 

15  Class  Differences  in  WST  Performance  of  Search 

Procedures .  32 

16  Class  Differences  in  ASW  Performance  of  Search 

Procedures .  32 

17  Crew:  Search  Phase  Efficiency .  34 

18  Crew:  Localization  Phase  Procedures .  35 

19  Individual:  Navigator .  37 

20  Navigator  WST  Performance  as  a  Function  of 

Training  Schedule  and  Class .  39 


MAVTRAEQUIPCEiN  70-C-0258-2 


LIST  OF  ILLUSTRATIONS  (  Coat  ) 


Figure  Page 

21  Navigator  ASW  Performance  as  a  Function  of 

Training  Schedule  and  Class .  39 

22  Individual:  Jezebel  Operator  Annotation  Accuracy 

and  Completeness  .  . . . .  43 

23  Individual:  Radar /MAD  Operator  Radar  Log 

Entries  .  45 

24  Individual:  Radar /MAD  Operator  Performance  of 

MAD  Procedures . 46 

25  Individual:  Radar /MAD  Operator  MAD  Tape 

Annotations . 46 

26  Trainer  Evaluation:  Task  Environment  Realism 

(N  =  2,  9,  7,  5) .  49 

27  Trainer  Evaluation:  Task  Performance  Realism 

(N  =  2, 8, 6, 4) .  51 

28  Trainer  Evaluations:  Gener^.  Training  Effective¬ 
ness  (N  =  3,  9,  7,  5) .  54 

29  Trainer  Evaluations:  Learning  Objective  Training 

Effectiveness  (N  =  2,  9,  7,  5) . 56 


v 


NAVTRAEQUIPCEN  70-C-0258-2 


LIST  OF  TABLES 

Table  Page 

1  Study  One  Population  Description .  5 

2  Study  Two  Description .  22 

3  Analysis  of  Variance  Summary  for  ASW  Training 

Effectiveness  for  Navigation  Stabilization  Procedures.  ...  28 

4  Analysis  of  Variance  Summary  for  Transfer  of  Navigation 

Stabilization.  Procedure  WST  Training  to  ASW  Training  ,  .  28 

5  Analysis  of  Variance  Summary  for  ASW  Training 

Effectiveness  for  Navigation  Stabilization  Accuracy .  31 

6  Analysis  of  Variance  Summary  for  WST  Training 

Effectiveness  for  Search  Phase  Procedures .  31 

7  Analysis  of  Variance  Summary  for  ASW  Training 

Effectiveness  for  Search  Phase  Procedures  . .  33 

8  Analysis  of  Variance  Summary  for  WST  Training 

Effectiveness  for  Localization  Phase  Procedures .  36 

9  Analysis  of  Variance  Summary  for  WST  Training 

Effectiveness  for  Navigator  Students  in  First  Class .  40 

10  Analysis  of  Variance  Summary  for  WST  Training 

Effectiveness  for  Navigator  Students  in  Second  Class  ....  40 

11  Analysis  of  Variance  Summary  for  ASW  Training 

Effectiveness .  41 

12  Analysis  of  Variance  Summary  for  Transfer  of 

Navigator  WST  Training  ASW  Training . .  41 

13  Analysis  of  Variance  Summary  for  WST  Training 
Effectiveness  for  Jezebel  Operator  Annotation 

Performance .  44 

14  Analysis  of  Variance  Summary  for  ASW  Training 
Effectiveness  for  Jezebel  Operator  Annotation 

Performance .  44 

15  Average  Instructor  Evaluations  of  the  Realism  of 

Training  Device  and  Aircraft  Training  Environment .  48 


NAVTRAEQUIPCEN  70-C-0258-2 


LIST  OF  TABLES  (  Cont  ) 

Table  Page 


16  Comparison  of  Trainer  Task  Environment  Realisms 
when  Evaluated  Against  Surface  Vessel  Training 

Flights  Versus  Submarine  Training  Flights .  50 

17  Comparison  of  Task  Performance  Realisms  in  the 

Trainer  when  Evaluated  Against  Surface  Vessel  Training 
Flights  Versus  Submarine  Training  Flights .  52 

18  Comparison  of  General  Training  Effectiveness 
Evaluations  for  the  TACCO  and  Navigator-  Positions 

(N  =  5) .  55 

19  Comparison  of  Learning  Objective  Training 
Effectiveness  Evaluations  for  the  TACCO  and 

Navigator  Positions  in  the  Trainer .  58 

20  Average  Instructor  Evaluations  of  the  Effectiveness 

of  Training  Device  and  Aircraft  Training  Environ¬ 
ments  for  Alternative  Training  Phases .  59 

21  Comparison  of  Training  Device  and  Aircraft  Training 

Effectiveness  Evaluation  for  the  TACCO  and  Navigator 
Positions  During  Alternative  Training  Phases  (N  =  3).  .  .  00 

22  Overall  Effectiveness  of  Training  Environments  per 

Training  Phase  (unweighted  means) .  6l 

23  Comparison  of  the  P3A/B  and  CATCC  Trainer  with 

Respect  to  Averaged  Task  Environment  Realism,  Task 
Performance  Realism,  and  Training  Effectiveness 
Characteristics  Evaluations .  68 

24  Comparison  of  the  S2E,  P3A/R  and  CATCC  Team 
Trainer  Evaluations  with  Respect  to  Specific  Realism 

and  Effectiveness  Questions .  68 


vii/ viii 


NAVTRAEQUIPCEN  70-C-0258-2 


SECTION  I 
INTRODUCTION 


PURPOSE  OF  THE  STUDY 

This  report  describes  the  results  o£  experiments  performed  to  evaluate  the 
training  effectiveness  of  the  Navy  2F69B  Weapon  System  Trainer.  This  eval¬ 
uation  effort  was  an  extremely  ambitious  undertaking  for  two  reasons.  First, 
the  trainer  is  designed  to  provide  training  for  the  entire  spectrum  of  flight  and 
ASW  tactics  that  can  be  employed  by  the  PiA  (and  B)  ASW  aircraft.  Second,  a 
principal  goal  of  the  study  effort  was  to  demonstrate  that  the  device  provides 
effective  training  for  both  crews  and  individual  crew  members. 

GENERAL  DESCRIPTION  OF  THE  2F69B  TRAINER 

The  weapon  system  trainer  for  the  P3A  and  B  aircraft  is  housed  in  two 
semitrailers.  Each  trailer  is  40  feet  long,  10  feet  wide,  and  12-1/2  feet  high. 
One  trailer  contains  the  Operational  Flight  Trainer  (OFT),  and  the  other  con¬ 
tains  the  Weapon  System  Trainer  (WST).  The  WST  is  also  referred  to  as  the 
Antisubmarine  Warfare  (ASW)  Tactics  Trainer.  The  OFT  and  WST  segments 
of  the  trainer  can  be  operated  concurrently  to  simulate  the  entire  ASW  mission 
of  the  P3  aircraft  or  independently  to  exercise  the  separate  skills  required  for 
aircraft  operation  or  submarine  detection,  tracking,  and  destruction. 

The  OFT  portion  of  the  trainer  is  used  to  train  student  pilots,  copilots,  and 
flight  engineers  on  both  normal  and  emergency  aircraft  operating  procedures, 
and  crew  coordination  procedures  during  ASW  operations.  The  WST  simulates 
the  five  ASW  operator  stations  of  the  P3A  aircraft.  These  are: 

a.  Jezebel  operator 

b.  Tactical  coordinator  (TACCO) 

c.  Navigator 

d.  Julie /ECM  operator 

e.  Radar/MAD  operator. 

The  WST  trailer  also  contains  instructor  consoles  and  equipments  which  are 
utilized  to  simulate  a  wide  variety  of  tactical  situations.  Computer  equipment 
in  the  WST  records  the  effects  of  operator  performance  at  each  of  the  crew 
stations.  Indicators  located  at  each  crew  position  are  "slaved"  to  instructor 
monitoring  indicators  to  enable  the  instructors  to  evaluate  ongoing  operator 
performance.  If  an  instructor  wishes,  he  can  "freeze"  simulated  conditions 
at  any  time  during  the  conduct  of  a  training  problem.  With  the  simulator 
"frozen,"  the  instructor  can  discuss  and  correct  erroneous  performance,  and 
either  reset  the  problem  to  a  new  set  of  initial  conditions,  or  continue  the 
problem  from  the  point  at  which  it  was  frozen. 
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COMPARISON  OF  OFT-WST  SEGMENTS  FOR  EVALUATION  PURPOSES 

Early  in  the  study  it  was  thought  that  the  flight  deck  or  OFT  portion  of  the 
trainei  would  be  worthwhile  to  evaluate.  However,  conversations  with  flight 
engineers  and  pilot  instructors  from  the  training  squadron  in  which  the  study 
was  to  be  performed  pinpointed  a  number  of  difficulties  inherent  in  complet¬ 
ing  such  an  evaluation.  These  were  as  follows: 

a.  The  OFT  was  used  to  teach  operating  procedures  rather  than  flight 
performance. 

b.  The  performance  of  flight  engineers  is  difficult  to  assess  because 
it  is  a  complex  function  of  pilot,  copilot,  and  flight  engineer  inter¬ 
action.  The  flight  engineer  seldom  initiates  an  action  in  the  simu¬ 
lator  unless  he  receives  instructions  from  the  pilot  or  copilot. 

c.  Flight  trainees  consist  primarily  of  transition  students  and  students 
undergoing  "refresher"  training. 

d.  There  was  no  means  for  obtaining  an  automated  record  of  simulator 
performance  for  the  flight  engineers  or  pilots. 

By  comparison,  the  WST  portion  of  the  trainer  appeared  likely  to  provide 
an  environment  that  was  more  conducive  to  experimentation.  For  example, 

a.  The  ASW  student  group  was  largely  composed  of  students  lacking 
previous  ASW  experience. 

b.  Performance  measures  descriptive  of  both  crew  and  individual  crew 
member  performance  was  available  for  each  of  the  crew  stations. 

c.  Alphanumeric  and  graphic  readouts  of  deviations  from  "real  world" 
conditions  were  available  at  instructor  stations. 

d.  Hard  copy  output  of  student  performance  was  available  in  the  form 
of  event  logs  at  each  operator  station  and  also  in  the  form  of  Julie 
and  Jezebel  tapes. 

e.  WST  instructors  agreed  to  participate  in  the  data  collection  portion 
of  the  evaluation  effort. 

f.  In  the  WST,  the  experimenter  was  to  be  allowed  to  control  the  com¬ 
plexity  levels  to  be  introduced  in  succeeding  training  sessions. 
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On  the  basis  of  the  comparison  of  the  operating  environments  existing  in  the 
OFT  and  WST  training  areas,  it  was  concluded  it  would  be  more  feasible  to 
complete  a  meaningful  evaluation  of  training  effectiveness  in  the  WST  section 
of  the  2Fb9B  trainer. 


The  evaluation  of  the  WST  was  accomplished  by  means  of  two  separate  pro¬ 
ject  experiments.  Each  experiment  is  described  in  detail  in  the  material  that 
follows.  The  descriptions  include  a  delineation  of  the  subject  population,  ex¬ 
perimental  design,  study  measures,  and  experimental  results.  Conclusions 
and  recommendations  extracted  from  each  study  are  collected  together  into  a 
single  discussion. 

In  addition  to  the  2F69B  evaluation,  a  small  cockpit  procedures  trainer, 
utilized  as  a  classroom  teaching  aid  was  evaluated  from  a  human  engineering 
viewpoint  to  determine  if  it  could  be  improved.  The  results  of  that  evaluation 
appear  in  Appendix  A  to  this  report. 
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SECTION  II 
EXPERIMENT  ONE 


INTRODUCTION 

Experiment  One  focused  on  ASW  operator  performance  as  it  occurred  in  the 
simulator.  The  intent  was  to  determine  whether  operator  performance  in  terms 
of  accuracy  and  efficiency  improved  as  the  student  progressed  through  five  WST 
training  sessions. 

EXPERIMENTAL  DESIGN 

SUBJECT  POPULATION.  Data  were  collected  on  two  classes  of  students  com¬ 
posed  of  six  and  seven  crews  respectively.  The  experimenters  were  unable  to 
control  the  number  of  students  being  trained  during  the  experimental  sessions; 
consequently,  each  crew  in  both  classes  did  not  have  an  equal  number  of  stu¬ 
dents  assigned  to  each  ASW  position.  The  range  and  average  number  of 
students  per  position  per  crew  appear  in  table  1. 


TABLE  1.  STUDY  ONE  POPULATION  DESCRIPTION 


Position 

First  Class 

Second  Class 

Total  Number 
of  Students 

TACCO 

1.7 

2.  0 

24 

1  -  2 

2 

Navigator 

1.8 

1.9 

24 

1-2 

1  -  3 

Jezebel 

1.3 

0.6 

12 

1  -  2 

0  -  2 

Julie /ECM 

1.5 

0.0 

9 

1  -  2 

0 

R.adar /MAD 

1.0 

0.  3 

8 

0  -  2 

0  -  1 

STUDY  METHODS 

Although  the  WST  sessions  varied  with  respect  to  mission  phase  emphasized 
and  the  ASW  techniques  utilized,  the  general  ASW  search,  localize,  and  destroy 
mission  piofile  was  employed  throughout  the  program  and  several  mission 
functions  and  activities  were  performed  during  two  or  more  WST  sessions.  A 
repeated  measures  design  with  subjects  serving  as  their  own  controls  was  used 
to  neutralize  the  possible  effects  of  individual  student  differences. 
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STUDY  MEASURES 

Data  were  collected  on  several  objective  measures  of  crew  and  individual 
performance.  Crew  measures  describe  the  accuracy  and  efficiency  with  which 
each  ASW  mission  phase  was  <.  impleted  by  those  crew  members  responsible  for 
that  phase.  Error  and  time  data  were  collected  and  converted  into  accuracy  and 
efficiency  measures  by  subtraction  from  a  constant  maximum  value  (e.g. , 
accuracy  =  constant  -  error).  As  a  result,  subjects  achieved  maximum  scores 
when  they  completed  a  mission  task  with  zero  error  and  minimum  time.  The 
emphasis  was  on  providing  meaningful  descriptions  of  performance  change. 

Individual  operator  measures  generally  describe  the  percentage  of  possible 
log  entries  and  sensor  record  annotations  accurately  completed  by  the  student. 
The  measures  of  crew  and  individual  performance  are  defined  in  greater  detail 
in  the  following  paragraphs. 

CREW  PERFORMANCE  MEASURES 

NAVIGATION  STABILIZATION.  This  task,  a  coordinated  effort  by  the  TACCO, 
Navigator,  and  Patrol  Pilot  in  Command  (PPC)  is  an  important  prerequisite  to 
successful  completion  of  the  localization  and  attack  mission  phases.  Stabiliza¬ 
tion  of  the  navigation  systems  (Nav  Stab)  establishes  an  accurate  North/South 
orienta.cion  or  relative  bearing.  This  relative  bearing  is  maintained  during 
subsequent  operations  by  calculating  buoy  drift  rates.  A  successful  Nav  Stab 
must  be  accomplished  if  the  crew  is  to  be  able  to  maintain  a  valid  match  be¬ 
tween  the  electronic  world,  used  by  the  TACCO  for  problem  resolution,  and 
the  real  world.  An  accuracy  measure  was  obtained  on  this  activity  which  re¬ 
flected  the  degree  of  success  with  which  the  crew  established  the  real  world 
position  and  drift  rate. 

SEARCH  PHASE.  The  crew  is  responsible  for  detecting,  locating,  and  iden¬ 
tifying  targets  of  interest.  During  the  search  phase  of  the  ASW  mission,  buoy 
patterns,  ECM  triangulation,  or  radar  run-ins  can  be  used,  but  buoy  patterns 
consistently  receive  the  greatest  emphasis  throughout  the  five  WST  sessions. 
Therefore,  the  crew  members  primarily  measured  by  Search  Accuracy  and 
Efficiency  scores  include  the  TACCO,  Navigator,  Jezebel  operator  and  PPC,  - 
The  search  phase  is  generally  concluded  by  the  dropping  of  a  datum  buoy  to 
mark  the  target's  estimated  position.  Accuracy  scores  for  this  phase  were 
based  on  the  difference  between  the  estimated  and  actual  target  position.  Effi¬ 
ciency  scores  were  derived  from  the  time  taken  to  complete  this  phase. 

LOCALIZATION  PHASE.  During  localization  the  ASW  crew  may  employ  any, 
or  all,  of  several  procedures  and  equipments  to  refine  their  estimate  of  the 
position  and  operating  parameters  of  the  target.  The  tactics  that  can  be 


It  should  be  noted  that  the  pilot  measures  obtained  in  the  ASW  section  of  the 
trainer  are  meaningless  The  pilot  is  seated  at  a  console  with  only  those  con¬ 
trols  needed  to  maintain  heading,  airspeed,  and  altitude.  These  controls  in  no 
way  represent  the  actual  controls  required  to  accomplish  piloting  tasks.  In 
effect,  the  pilot's  role  in  the  ASW  section  of  the  trainer  is  primarily  to  learn  the 
appropriate  voice  and  crew  coordination  procedures. 
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employed  are:  CODAR,  Julie,  Active  Buoy,  and  MAD.  The  CODAR  pattern  is 
usually  the  first  tactic  employed  followed  by  a  Julie  or  Active  Buoy  pattern. 
Magnetic  anomalie  detection  (MAD)  procedures  may  be  employed  at  any  time, 
but  they  usually  accompany  or  follow  the  Active  pattern.  The  Localization 
Efficiency  measure  described  the  duration  of  the  CODAR  pattern  and  assessed 
the  performance  of  the  TACCO,  Navigator,  and  Jezebel  operator.  Accuracy 
scores  were  derived  from  the  difference  existing  in  the  distance  between  the 
target's  actualposition  and  the  marker  buoy  position  or  the  weapons  store  drop 
position,  whichever  was  appropriate. 

ATTACK  PHASE.  For  measurement  purposes,  the  attack  phase  was  considered 
to  begin  when  entry  into  a  Julie  or  Active  Buoy  pattern  occurred,  and  to  end  with 
a  successful  kill  or  problem  termination.  Efficiency  measures  were  derived 
from  this  time  interval.  Attack  accuracy  measures  describe  the  deviation  be¬ 
tween  weapons  stores  dropped  and  the  criterion  distance  from  the  target. 

INDIVIDUAL  PERFORMANCE  MEASURES 

NAVIGATOR.  During  the  course  of  the  ASW  mission,  the  Navigator  performs 
both  a  computational  and  a  bookkeeping  function.  As  directed  by  the  TACCO, 
he  develops  a  hardcopy  trace  of  target  position,  heading,  speed,  etc.,  based 
on  data  inputs  from  the  other  crew  members.  This  dead  reckoning  trace  (DR.T) 
provides  the  TACCO  with  one  criterion  on  which  to  prosecute  the  mission.  In 
addition,  the  Navigator  is  responsible  for  documenting  the  mission  in  his  Navi¬ 
gator's  Log. 

One  facet  of  Navigator  proficiency  is  his  ability  to  keep  abreast  of  and  re¬ 
cord  data  inputs  he  receives  during  each  phase  of  the  mission.  To  score  Navi¬ 
gator  proficiency,  items  he  processed  on  his  DRT  and  log  were  compared  with 
the  stimuli  it  could  be  assumed  he  received  from  each  of  the  other  ASW  crew 
positions.  A  percentage  score  was  computed  which  described  the  accuracy 
and  completeness  of  his  annotations. 

JEZEBEL  OPERATOR.  During  the  search  and  localization  phases  of  the  ASW 
mission,  the  Jezebel  operator  participates  in  LOFAR  search  and  CODAR  local¬ 
ization.  During  both  of  these  activities  he  is  responsible  for  recognizing,  inter¬ 
preting,  reporting,  and  annotating  sensor  data  and  other  stimuli  appearing  on 
his  synchronization  chart.  At  the  conclusion  ol  each  trainer  session,  the  syn¬ 
chronization  charts  were  scored  by  noting  the  accuracy  and  completness  of  the 
mission  annotations.  The  operator's  score  was  the  percentage  of  possible  an¬ 
notations  correctly  accomplished  by  the  operator. 

JUL1E/ECM  OPERATOR.  During  the  ASW  mission,  the  Julie/ECM  operator 
serves  two  functions.  The  first  occurs  during  the  search  phase  wherein  he 
monitors  and  reports  electronic  signals  emanating  from  suspected  targets. 

This  function  is  addressed  very  infrequently  during  the  five  trainer  sessions. 

The  second  function  occurs  during  Julie  localization  wherein  the  operator  mon¬ 
itors  and  annotates  a  paper  tape  on  which  target  returns  are  traced.  In  con¬ 
junction  with  his  Julie  duties,  the  operator  also  monitors  and  annotates  taped 
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responses  resulting  from  Active  Buoy  detonations.  Here  he  is  serving  as  a 
backup  to  the  AQA-1  Operator.  The  student's  annotated  tapes  were  reviewed 
by  instructor  personnel  at  the  conclusion  of  each  session  in  which  he  performed. 
The  resulting  Julie  score  was  the  percent  of  possible  annotations  correctly  made 
by  the  student. 

RADAR/MAD  OPERATOR.  This  position  is  responsible  for  operating  the  radar 
and  MAD  equipments  as  directed  by  the  TACCO.  The  MAD  gear  is  used  more 
frequently  across  the  five  WST  sessions  than  the  radar  and  provided  a  better 
means  to  obtain  evaluative  measurements.  During  MAD  operations,  the  stu¬ 
dent  is  responsible  for  monitoring,  interpreting,  reporting,  and  annotating  the 
signals  appearing  on  the  MAD  tape.  The  student's  score  was  the  percent  of 
possible  annotations  correctly  accomplished. 

DATA  COLLECTION  METHODS.  The  evaluation  data  were  collected  by  Bunker 
Ramo  personnel,  assisted  by  WST  instructors.  Information  for  the  team  mea¬ 
surement  data  were  obtained  by  monitoring  the  crew  communications,  student 
and  instructor  extra-channel  communications,  the  mission  plot  retained  on  the 
WST  optical  pattern  display,  target  to  buoy  distances,  and  digital  readouts  lo¬ 
cated  at  the  instructor  station.  Annotation  data  were  scored  by  Navy  instructor 
personnel  to  ensure  the  necessary  technical  understanding  and  expert  judgment 
required  for  completing  the  operator  tasks  at  each  position.  These  data  were 
collected  for  two  student  classes  in  order  to  increase  the  overall  N. 

DATA  ANALYSIS  METHODS.  Graphic  analysis  was  selected  to  describe  the 
effects  of  repeated  use  of  the  trainer  on  student  performance  levels.  This  meth¬ 
od  was  deemed  the  most  appropriate  because  (1)  the  aspects  of  performance 
measured  were  ones  practiced  more  than  once  by  the  students  during  the  train¬ 
ing  program,  and  (2)  the  sample  sizes  were  generally  too  small  to  be  satisfac¬ 
torily  evaluated  with  quantitative  statistical  methods.  Two  other  factors  were 
also  expected  to  affect  performance  levels:  individual  and  team  differences, 
and  WST  problem  differences.  These  factors  were  controlled  to  the  extent 
possible . 

Individual  and  crew  differences  were  controlled  by  using  repeated  measures 
across  the  five  WST  sessions  for  each  of  the  measures  of  student  performance 
(the  one  exception  to  this  will  be  noted  in  the  Results  section).  Fairly  complete 
control  oi  individual  and  crew  dillerences  was  possible  lor  data  collected  on  the 
first  class  because  accurate  information  on  crew  member  identification  was 
available.  During  data  collection  for  the  second  class,  the  instructors  did  no: 
always  maintain  an  accurate  record  of  which  team  member  was  being  trained 
at  a  position  during  each  WST  session.  From  a  careful  examination  of  the  data 
collection  sheets,  it  appeared  that  if  more  than  one  student  for  any  one  position 
was  included  on  a  crew,  say  crew  M,  then  crew  M(l)  performed  on  WSTs  1,  3, 
and  5,  while  crew  M(2)  performed  on  WSTs  2  and  4. 

To  ensure  that  WST  instructors  employed  a  standardized  set  of  instructions 
during  the  evaluation,  the  effects  of  problem  variations  on  study  goals  were 
discussed  with  instructor  personnel.  The  instructors  were  requested  before 
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and  during  the  study  to  adhere  to  the  WST  syllabus,  and  ensure  that  the  initial 
problem  condition  for  any  particular  WST  (e.g.,  number  of  targets,  sea  and 
air  environmental  conditions)  was  the  same  for  all  crews.  Data  descriptive 
of  problem  and  environmental  condilions  wore  collected  throughout  each  WST 
session.  These  descriptions  were  reviewed  for  each  set  of  measures  for  each 
of  the  five  WST  sessions  to  ensure  omparable  data. 

STUDY  ONE  RESULTS 

The  empirical  results  of  Study  One  arc  discussed  and  presented  graphically 
in  the  pages  that  follow.  Both  crew  and  individual  performance  curves  support 
the  learning  hypothesis.  A  trend  toward  improved  performance  as  a  function 
of  repeated  trainer  use  was  found  in  all  cases. 

The  organization  of  the  data  sample  was  such  that  it  was  feasible  to  use  the 
independent  variable  axis  to  describe  WST  session  number,  thus  permitting 
presentation  of  the  data  within  a  training  schedule  and  program  context.  It  did 
not  appear  that  any  one  of  the  individual  WST  sessions  had  any  remarkable  or 
consistent  effects  on  performance  over  and  beyond  the  effects  that  could  be  ex¬ 
pected  to  result  from  repeated  performance. 

CREW  PERFORMANCE 

NAVIGATION  STABILIZATION  ACCURACY  The  performance  curves  presented 
in  figure  \  indicate  a  consistent  tendency  toward  improved  Nav  Stab  Accuracy  as 
the  crews  progress  through  their  training  program.  It  will  be  noted  that  two 
performance  curves  appear  in  this  and  most  of  the  subsequent  figures.  There 
are  no  distinctions  between  these  figures  other  than  that  they  represent  unique 
sets  of  crews  (one  set  of  10  crews  and  another  set  of  2  crews  appear  in  figure 
1).  Two  curves  were  usually  necessary  to  ensure  repeated  measures  on  each 
performance  curve  across  the  WST  sessions.  It  is  of  interest  to  observe  that 
although  there  is  considerable  difference  between  the  initial  performance  levels 
displayed  by  the  two  curves,  these  differences  are  minimal  for  the  second  Nav 
Stab  performances.  This  is  probably  a  direct  result  of  the  fact  that  some  crews 
exhibited  procedural  and  coordination  problems  during  the  first  attempt  at  Nav 
Stab,  which  were  subsequently  corrected  by  instruction  and  practice.  It  is  sug¬ 
gested  that  both  of  the  crews  peifoiming  their  initial  Nav  Stab  on  WST  2  had 
procedural  and/or  coordination  prob'ems. 

SEARCH  PHASE  ACCURACY  AND  EFFICIENCY.  The  accuracy  and  efficiency 
of  search  phase  performance  as  the  crews  progressed  through  training  is  de¬ 
scribed  in  figures  2  and  3.  Again,  a  consistent  trend  toward  improved  per¬ 
formance  can  be  observed.  The  slope  differences  between  the  curves  in  Fig¬ 
ure  3  are  Interesting  and  may  suggest  that  amount  of  training  transfer  for  this 
performance  criterion  was  affected  by  duration  of  the  time  interval  between 
performances.  The  crew  Ns  are  too  small,  however,  to  be  more  than 
suggestive. 


9 


ACCURACY 


NAVTRAEQUIPCEN  70-C-0258-2 


12  3  4  S 


WS1  SESSION 


Figure  1.  Crew:  Navigation  Stabilization  Accuracy 
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WST  SESSION 

Figure  2.  Crew:  Search  Phase  \ccuracy 
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WST  SESSION 

Figure  3.  Crew:  Search  Phase  Efficiency 
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LOCALIZATION  PHASE  ACCURACY  AND  EFFICIENCY.  Figures  4  and  5 
describe  accuracy  and  efficiency  changes  in  crew  performance  during  the 
localization  phase.  The  evidence  indicates  that  although  performance  ac¬ 
curacy  may  be  poorer  for  the  first  localization  phase  performed,  it  quickly 
tends  to  reach,  and  remain  near  the  values  of  275  and  280.  Unfortunately, 
the  accuracy  curve  for  N  --  4  could  not  be  described  tor  WST  2  because  of 
the  incomplete  set  of  repeated  measures  data.  In  can  be  said,  however,  that 
those  crews  initially  performing  at  a  lower  level  all  improved  to  the  extent 
of  fluctuating  within  the  above-mentioned  interval. 

Both  localization  phase  efficiency  curves  indicated  performance  improve¬ 
ment  as  a  function  of  repeated  trainer  use.  Again,  the  learning  hypothesis 
appears  to  be  supported. 

ATTACK  PHASE  ACCURACY  AND  EFFICIENCY.  Curves  describing  crew 
performance  accuracy  and  efficiency  during  the  attack  phase  are  presented  in 
figures  6  and  7.  When  and  how  the  attack  phase  is  performed  during  training 
is  a  function  of  not  only  syllabus  requirements,  but  also  the  manner  in  which 
any  particular  ASW  mission  progresses  and  the  crew's  training  needs  are 
perceived  by  instructor  personnel.  As  a  result,  an  adequate  sample  of  com¬ 
parable  repeated  measures  could  not  be  obtained  for  this  phase.  Figures  6 
and  7  provide  the  best  available  estimates  of  team  performance  during  WST 
sessions  3,  4,  and  5  and  appear  to  support  the  learning  hypothesis. 

INDIVIDUAL  PERFORMANCE  MEASURES 

NAVIGATOR  ANNOTATION  ACCURACY  AND  COMPLETENESS.  Performance 
curves  desi  ribing  the  Navigator's  ability  to  correctly  process  all  the  informa¬ 
tion  inputs  he  receives  during  the  mission  are  presented  in  figure  8.  It  will 
be  observed  that  the  curves  display  a  consistent  tendency  for  improved  perfor¬ 
mance  in  WSTs  3,  4,  and  5,  as  compared  co  the  first  two  WST  sessions.  It 
appears  that  the  Navigator's  initial  ability  to  process  information  inputs  is 
improved  by  repeated  performance. 

JEZEBEL  OPERATOR  ANNOTATION  ACCURACY  AND  COMPLETENESS. 
Figure  9  describes  the  changes  in  Jezebel  Operator  capability  to  correctly 
and  completely  annotate  the  information  appearing  on  his  synchronization 
chart  during  WST  training.  Again,  the  curves  support  the  learning  hypothe¬ 
sis.  The  differences  between  the  curves  with  respect  to  WSTs  1  and  2  appear 
to  have  resulted  from  the  effects  of  chance  on  small  sized  samples  of  students 
with  varying  initial  capabilities. 

JULIE/ECM  OPERATOR  ANNOTATION  ACCURACY  AND  COMPLETENESS. 
The  change  in  Julie/ECM  annotation  proficiency  over  WSTS  3  and  5  is  de¬ 
scribed  in  figure  10.  The  change  is  a  positive  one  and  supports  the  hypothe¬ 
sis  that  performance  improves  as  a  function  of  repeated  performance  in  the 
trainer. 
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Figure  9.  Individual:  Jezebel  Operator  Annotation 
Accuracy  and  Completeness 


NAVTRAEQUIPCEN  70-C-0258-2 


Figure  10.  Individual:  Julie/ECM  Operator  Annotation  Accuracy 
and  Completeness  (N  =  6) 
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RADAR /MAD  OPERATOR  ANNOTATION  ACCURACY  AND  COMPLETENESS. 
Proficiency  in  accurately  and  completely  annotating  the  MAD  tape  as  a  function 
of  WST  session  number  is  described  in  figure  11.  Although  the  small  student 
N  allows  only  a  very  tentative  evaluation,  it  again  appears  that  repeated  use 
of  the  trainer  benefits  performance  of  this  task. 
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SECTION  III 
EXPERIMENT  TWO 


INTRODUCTION 

In  the  second  experiment  in  this  series,  the  data  collection  effort  was  ex¬ 
tended  to  the  airborne  environment.  The  purpose  of  this  second  study  was  to 
demonstrate  the  extent  to  which  simulator  training  transferred  to  the  inflight 
ASW  training  environment.  An  ideal  experimental  design  for  this  second 
effort  would  have  been  to  provide  simulator  training  to  one  group  of  subjects, 
not  provide  it  to  a  second  group,  train  both  groups  in  the  airborne  environ¬ 
ment,  and  later  check  the  performance  of  both  the  experimental  and  control 
group  in  their  operational  ASW  squadrons.  Although  the  experimenters  were 
allowed  to  modify  the  training  schedule  slightly,  they  were  unable  to  achieve 
the  ideal  experimental  design.  Instead,  an  intermediate  approach  was  author¬ 
ized  by  the  training  squadron.  In  the  authorized  approach,  the  experimental 
procedure  discussed  above  was  to  be  partly  applied,  in  that  at  least  two  groups 
cf  subjects  were  to  be  introduced  (excluding  the  last  training  session)  to  a 
problem  in  the  airborne  environment  first.  They  were  to  then  proceed  to  the 
simulator.  This  is  the  reverse  of  the  normal  training  procedure.  Due  to 
weather  problems  and  class  size,  the  design  would  not  be  employed  past  the 
first  training  session. 

In  the  planning  stage  of  study  design,  the  intent  was  to  test  airborne  ASW 
performance  against  actual  submarine  targets.  Unfortunately,  the  task  of 
scheduling  submarines  to  coincide  with  training  flight  timetables  proved  to  be 
out  of  the  question.  Thus,  airborne  performance  was  tested  as  the  ASW  op¬ 
erators  worked  against  targets  of  opportunity  which  generally  proved  to  be 
surface  ships  from  various  merchant  fleets.  Obviously,  the  acoustical  returns 
from  such  vessels  would  be  less  than  optimum  for  training  submarine 
procedures. 

SUBJECT  POPULATION 

Data  were  collected  on  two  classes  of  students  composed  of  4  and  7  crews, 
respectively.  Again,  as  in  Study  One,  none  of  the  crews  in  either  class  were 
constituted  of  one  student  for  each  position.  The  range  and  average  number 
of  students  per  position  per  crew  appear  in  table  2. 

EXPERIMENTAL  DESIGN 

Squadron  training  schedules  were  modified  to  provide  experimental  and 
control  student  groups  for  a  limited  transfer  of  training  evaluation.  The  sched¬ 
ule  for  Wings  A  and  £  of  two  classes  were  as  follows: 

Class  One:  Wing  A:  WST  l/ASW  1/WST  2/ASW  2/WST  3/ASW  3/WST 
4  ASW  4/ WST  5 /ASW  5 

Wing  B:  ASW  1/WST  l/ASW  2/WST  2/WST  3/ASW  3/WST 
4/ASW  4 /WST  5 /ASW  5 
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TABLE  2.  STUDY  TWO  DESCRIPTION 


Position 

First  Class 

Second  Class 

Total 

Number  of  Students 

TACCO 

1.8 

1 . 6 

18 

1-2 

1-2 

Navigator 

2.0 

2.0 

22 

2 

1-3 

Jezebel 

1.3 

0.  7 

10 

0-3 

0-1 

Julie/ECM 

0.0 

0.  0 

0 

Radar  /MAD 

1.3 

0.  3 

7 

1-2 

0-1 

Class  Two:  Wing  A:  WST  l/WST  2/ASW  l/WST  3/ASW  2/WST  4/A SW 
3/ASW  4/ WST  5/ASW  5 

Wing  B:  ASW  l/WST  l/WST  2/ASW  2/WST  3/ASW  3/WST 
4/ASW  4/ASW  5 /WST  5 

The  training  schedule  represented  by  Wing  A  of  Class  One  is  the  normal 
training  schedule.  The  training  schedule  for  Wing  A  of  Class  Two  was  to  have 
been  the  normal  schedule  but  was  modified  to  accommodate  a  hurricane.  For 
the  particular  transfer  of  training  hypothesis  evaluated  in  this  study,  both  A 
Wings  were  considered  members  of  the  same  group.  B  Wings  represent  the 
training  schedule  modifications  permitted  for  study  purposes. 

Two  transfers  of  training  hypotheses  were  considered:  (1)  that  prior  WST 
training  in  the  device  would  positively  affect  inflight  ASW  performance  and, 
conversely,  (2)  that  prior  ASW  inflight  training  would  positively  affect  subse¬ 
quent  WST  performance.  Wings  A  versus  B  constituted  the  experimental  and 
control  groups  for  these  hypotheses,  which  can  be  stated  as  follows: 

a.  Wings  A  will  perform  ASW  1  better  than  Wings  B 

b.  Wings  B  will  perform  WST  1  better  than  Wings  A 

It  will  be  noted  that  the  training  schedule  for  Wing  B  of  Class  Two  was  also 
modified  to  permit  an  evaluation  of  transfer  from  ASW  to  WST  5  and  vice  versa. 
Unfortunately,  insufficient  comparable  data  were  obtained  to  evaluate  these 
hypotheses. 
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As  Classes  One  and  Two  represented  independent  samples  of  students,  the 
possible  contribution  of  a  class  factor  was  evaluated  statistically  whenever 
sample  sizes  permitted. 

The  above  study  designs  can  be  summarized  as  follows.  The  design  for  the 
transfer  of  training  hypotheses  was  a  2  x  2  factorial  design  using  independent 
samples.  The  first  factor  consisted  of  the  experimental  and  control  groups. 
Wings  A  and  B,  where  the  experimental  group  was  that  Wing  which  had  had 
prior  training  in  the  alternative  training  environment  and  the  control  group 
was  that  Wing  which  had  not  had  any  prior  training.  The  second  factor,  when 
included,  was  Classes  One  and  Two. 

The  WST  and  ASW  learning  hypotheses  were  evaluated  with  a  repeated  mea¬ 
sures  design  for  the  factor  of  Training  Environment. 

Since  the  results  obtained  in  Study  One  did  not  indicate  that  individual  WST 
sessions  (e.g.,  WST  3)  had  distinctive  or  consistent  effects  on  performance 
variability,  it  was  decided  to  consider  anilysis  and  presentation  of  student 
performance  data  in  terms  of  the  number  of  sessions  performed  rather  than 
WST  session  number.  Further,  the  varying  number  of  students  included  in 
each  crew  for  each  position  made  it  necessary  to  display  student  performance 
data  against  an  abscissa  defined  in  this  manner  in  order  to  depict  possible 
learning  effects. 

STUDY  MEASURES 

Several  objective  measures  of  crew  and  individual  performance  were  col¬ 
lected  in  Study  Two  which  were  quite  similar  to  those  collected  in  Study  One 
and  shall  be  considered  identical  in  terms  of  general  definition.  The  differ¬ 
ences  are  primarily  the  result  of  different  measurement  procedures  (proce¬ 
dures  appropriate  for  both  training  environments  were  used  in  Study  Two> 
which  led  to  somewhat  different  sets  of  mission  phase  activities  being  assessed 
in  the  two  studies,  even  though  there  was  considerable  overlap  between  the  sets. 
Full  measure  definitions  or  distinctive  Study  Two  measure  characteristics  are 
given  below  only  where  the  general  definition  has  been  essentially  changed. 
Appendix  B  includes  the  criterion  scoring  forms  employed  in  Study  One. 
Appendix  C  contains  those  used  in  Study  Two. 

A  new  objective  measure  of  performance  was  collected  on  the  mission 
phases  and,  where  feasible,  on  the  individual  positions.  This  measure  as¬ 
sessed  the  team  and  the  operators  procedural  proficiency  in  terms  of  complete 
and  correct  performance.  Procedural  omissions  and  errors  represented  the 
difference  between  a  100%  procedures  score  and  the  percent  score  given. 

In  addition  to  objective  performance  measures,  data  collected  on  an  evalua¬ 
tive  set  of  measures  were  used  to  provide  a  basis  for  interpreting  the  objective 
data  and  for  recommending  modifications  and/or  courses  of  action  to  improve 
device  utility  and  effectiveness.  These  evr’uative  training  device  measures 
were  collected  by  use  of  a  questionnaire  (see  Appendix  D)  which  was  completed 
by  instructor  personnel  for  their  particular  position  (e.g.  ,  TACCO/Navigator 
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instructors  versus  Radar/MAD  instructors).  Instructors  answered  questions 
with  reference  to  only  their  own  position  or  the  crew  as  a  whole  and  specified 
when  they  were  giving  the  latter  type  of  answer.  The  questions  related  to 
specific  characteristics  of  the  training  device  and  their  relationship  to  training 
objectives,  learning  phases,  and  equipment  and  operator  functions. 

CREW  PERFORMANCE  MEASURES 

NAVIGATION  STABILIZATION.  Two  measures  of  Nav  Stab  performance  were 
obtained:  procedures  and  accuracy.  A  crew's  procedures  score  was  the  per¬ 
centage  of  procedures  that  should  have  been  performed  correctly.  Although 
the  accuracy  values  obtained  in  Studies  One  and  Two  are  not  directly  comparable, 
because  of  a  change  in  emphasis  on  Nav  Stab  methods  utilized,  the  general  defini¬ 
tion  of  the  Nav  Stab  accuracy  measure  remains  the  same. 

SEARCH  PHASE.  Three  measures  of  crew  search  phase  performance  were 
obtained:  procedures,  accuracy,  and  efficiency.  The  procedures  score  was 
again  the  percentage  of  procedures  correctly  performed  and  not  erroneously 
omitted.  The  accuracy  and  efficiency  measure  definitions  remain  essentially 
the  same. 

CREW  LOCALIZATION  PHASE.  A  measure  of  localization  phase  procedures 
was  obtained  and  is  defined  as  above.  Insufficient  data  were  collected  on  lo¬ 
calization  accuracy  and  efficiency  to  permit  evaluations  of  student  performance 
differences  or  changes. 

INDIVIDUAL  PERFORMANCE  MEASURES 

NAVIGATOR.  The  Navigator  measure  obtained  in  Study  Two  was  a  total  score 
which  assessed  not  only  the  accuracy  and  completeness  of  DRT  and  log  annota¬ 
tions,  but  also  adherence  to  procedures  (e.g.,  periodicity  of  stabilization 
store  mark  on  tops).  The  measures  applied  were  those  normally  used  by  the 
training  squadron  for  Navigator  assessment.  The  total  score  represents  the 
sum  of  three  weighted  percentages  (procedures,  tactical  log,  tactical  chart) 
where  the  maximum  possible  component  and  total  scores  are  100. 

JEZEBEL  OPERATOR.  In  both  Studies  One  and  Two,  the  Jezebel  operator's 
score  was  the  percentage  of  possible  synchronization  chart  annotations  cor¬ 
rectly  made  by  the  operator,  based  on  instructor  scoring  of  the  chart.  Be¬ 
cause  it  was  not  possible  to  have  the  charts  from  both  studies  evaluated  by 
the  same  instructor  personnel,  the  scores  obtained  in  the  two  studies  should 
not  be  considered  completely  comparable. 

JULIE/ECM  OPERATOR.  There  were  no  Julie/ECM  students  participating 
in  the  second  study. 

RADAR/MAD  OPERATOR  Measures  collected  on  this  position  during  the 
second  study  included:  radar  log  entries,  percent  complete;  MAD  procedures, 
percent  correctly  performed;  and  MAD  tape  annotations,  percent  correctly 
made.  Again,  because  it  was  not  possible  to  have  the  MAD  tapes  from  both 
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studies  evaluated  by  the  same  instructor  personnel,  the  scores  obtained  in 
the  two  studies  should  not  be  considered  directly  comparable. 

DATA  COLLECTION  METHODS.  The  data  were  collected  by  instructor  per¬ 
sonnel  using  forms  especially  developed  for  each  of  the  individual  sensor 
stations  and  for  the  team  as  a  vhole,  The  primary  emphasis  of  crew  mea¬ 
sures  was  on  the  TACCO/Navigator /PPC  Subteam.  Additional  mission 
descriptive  data  were  collected  in  the  device  WST  sessions  by  a  Bunker 
Ramo  project  member,  assisted  by  device  personnel.  Sensor  records, 
charts,  and  logs  were  scored  by  instructor  personnel  to  ensure  the  technical 
cognizance  and  judgment  required  for  each  position. 

DATA  ANALYSIS  METHODS.  Analysis  of  variance  was  used  to  evaluate  the 
transfer  of  training  and  learning  hypotheses  where  there  was  sufficient  data  to 
do  so.  T-test  or  two-factor  methods,  for  independent  or  repeated  measures, 
for  equal  or  unequal  Ns,  were  used  as  appropriate.  Graphic  analysis  was  per¬ 
formed  in  all  cases  to  evaluate  and  display  student  performance  variations  as 
a  function  of  repeated  performance. 

STUDY  TWO  RESULTS,  This  section  of  the  report  presents  the  objective 
performance  data  collected  in  Study  Two.  The  results  of  statistical  and  graphic 
analyses  are  presented  and  evaluated  within  the  observed  training  context.  A 
number  of  the  performance  curves  were  based  on  incomplete  replications  of 
the  sample  population.  Considerable  data  was  lost  due  to  incomplete  entries 
on  data  collection  forms.  These  scoring  deletions  occurred  when  the  instruc¬ 
tors  elected  to  ignore  data  collection  in  favor  of  providing  the  student  with 
directions.  An  effort  was  made  to  base  the  curves  on  the  most  complete  set 
of  replications,  or  repeated  measures,  possible  (e.g.  ,  if  the  only  measure 
obtained  on  an  operator  was  during  his  fourth  WST  performance,  this  datum 
did  not  contribute  to  the  estimation  of  WST  4  performance  level). 

In  general,  it  was  found  that  aircraft  ASW  performances  displayed  more 
consistent  learning  trends  and  higher  levels  of  performance  than  did  the  trainer 
WST  performances  and  that  only  hypotheses  of  transfer  from  the  trainer  to  the 
aircraft  were  supported.  I'liis  was  not  considered  surprising  considering  the 
following:  (1)  the  training  device  probably  provides  a  better  instructional  en¬ 
vironment  (due  to  lower  noise  levels  and  ability  to  freeze  the  problem  for 
discussion  purposes),  and  (Z)  the  compositions  of  most  of  the  student  crews 
during  this  study  (e.g,,  Crew  M  with  three  TACCOs,  two  Navigators,  two 
PPCs,  one  Jezebel  operator,  and  no  radar  operators;  Crew  N  with  one  TACCO, 
three  Navigators,  two  PPCs,  three  Jezebel  operators,  and  one  radar  operator; 
etc.  )  necessitated  training  two  or  more  teams  per  crew  within  a  training  pro¬ 
gram  and  time  schedule  designed  for  one  person  per  ASW  crew  position.  The 
latter  situation  forced  the  instructors  to  trade-off  students  so  as  to  provide  all 
students  with  some  exposure.  This  was  probably  better  accomplished  in  the 
-'nstructional  environment  of  the  trainer.  Even  so,  such  a  training  regimen 
could  be  expected  to  fall  short  of  providing  consistent  or  optimum  learning 
effects. 
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CREW  PERFORMANCE  MEASURES 

NAVIGATION  STABILIZATION  PROCEDURES  AND  ACCURACY.  Figure  12 
describes  WST  and  ASW  Nav  Stab  Procedure  performance  trends  based  on 
what  is  best  described  as  independent  samples.  Both  the  WST  and  ASW  curves 
depict  a  general  trend  to  improved  performance.  The  average  initial  perfor¬ 
mance  levels  vary  between  84  and  94%,  while  the  second  and  third  ASW  and 
third  and  fourth  WST  performances  vary  between  averages  of  98  and  100% 
procedurally  correct. 

Although  there  was  insufficient  repeated  measures  data  to  statistically  test 
the  WST  performance  change,  the  results  of  testing  the  ASW  performance 
changes  between  the  first  and  second  Nav  Stab  performance  were  significant 
at  the  0,01  level  (see  table  3);  i.  e.  ,  Nav  Stab  procedural  performance  did  im¬ 
prove  significantly  in  the  ASW  training  environment.  There  was  also  an  in¬ 
teraction  trend  between  training  and  class  (p<  0. 10)  which  resulted  from  the 
differing  initial  performance  levels  between  the  two  classes  (74%  versus  93%). 
The  difference  in  initial  performance  levels  could  have  resulted  from  many 
factors,  including  student  and  instructor  differences.  It  is  also  possible  that 
the  difference  could  have  resulted  from  modifications  made  in  a  prior  systems 
training  course.  The  class  with  an  initial  mean  performance  level  of  93%  had 
received  the  modified  course. 

The  hypothesis  that  prior  performance  in  the  WST  environment  would  bene¬ 
fit  initial  ASW  performance  was  tested  and  a  tendency,  significant  at  p  <  0.  10 
v.as  found  (see  table  4).  Despite  the  small  cell  Ns,  the  data  trends  were  con¬ 
sistent.  The  mean  initial  ASW  performance  value  for  TACCO/Navigator  teams 
who  had  performed  a  Nav  Stab  in  the  trainer  first  was  96%.  The  mean  initial 
ASW  performance  value  for  students  performing  a  Nav  Stab  for  the  first  time 
was  85%. 

Figure  13  describes  WST  and  ASW  Nav  Stab  Accuracy  and  is  based  on  re¬ 
peated  measures.  The  individual  Ns  for  the  WST  curve  are  4,  4,  1,  and  1 
and  1  for  Nav  Stab  pei formances  1,  2,  4,  and  5,  respectively.  The  first  and 
second  performance  values  for  the  one  team  providing  samples  of  fourth  and 
fifth  WST  Nav  Stab  performances  are  closely  represented  by  the  mean  curve. 

In  Study  One,  the  measure  of  Nav  Stab  Accuracy  was  an  average  of  the 
several  attempts  to  accurately  stabilize  the  navigation  system.  The  data  ob¬ 
tained  for  that  measure  displayed  a  consistent  learning  trend  during  WST 
training  (see  figure  1).  In  Study  Two,  the  obtainable  measure  of  Nav  Stab 
Accuracy  was  the  final  accuracy  with  which  the  system  was  stabilized.  This 
difference  in  measure  definition  may  be  responsible  for  the  different  type  of 
Nav  Stab  Accuracy  performance  curves  obtained  in  Study  Two  because  different 
aspects  of  behavior  were  being  measured.  The  instructors  intend  to  give  pri¬ 
mary  emphasis  to  Nav  Stab  instruction  the  first  time  it  is  performed  by  the 
students.  Consequently,  if  the  initial  attempts  at  system  stabilization  are  not 
considered  sufficiently  accurate,  the  procedures  may  be  repeated  as  necessary 
until  reasonable  accuracy  is  established.  In  subsequent  sessions,  the  integra¬ 
tion  of  Nav  Stab  into  the  mission  context  is  emphasized.  The  average  perfor¬ 
mance  levels  for  the  first  and  second  WST  and  ASW  performances  presented  in 
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—  —  WST  (Ns  4  TO  11 ) 
—  ASW  (Ns  -8  TO  11) 


12  3  4 

NAY  STABS,  NO.  PERFORMED 


NOTE:  These  curves  represent  best  estimates, 

based  on  independent  measures;  i.e.,  data 
are  not  controlled  witn  respect  to  crew 
identity. 

Figure  12.  Crew:  Navigation  Stabilization  Procedures 
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TABLE  3.  ANALYSIS  OF  VARIANCE  SUMMARY  FOR  ASW  TRAINING 
EFFECTIVENESS  FOR  NAVIGATION  STABILIZATION  PROCEDURES 


Source  of  Variation 

SS 

df 

MS 

F 

Between  Subjects 

A  (Classes) 

403. 79 

1 

403. 79 

3.  39 

Subjects  within  groups 

909. 54 

8 

119. 25 

Within  Subjects 

B  (ASW  Nav  Stabs) 

1302.  24 

1 

1302.  24 

11. 47** 

AB 

403. 70 

1 

403.  70 

3.  55* 

B  x  Subjects  within 
groups 

908. 54 

8 

113. 57 

VF  0.90  (1,8)  *  3.46 
**F  0.01  (1,  8)  =  11.3 


TABLE  4.  ANALYSIS  OF  VARIANCE  SUMMARY  FOR 
TRANSFER  OF  NAVIGATION  STABILIZATION  PROCEDURE  WST 
TRAINING  TO  ASW  TRAINING 


SS 

df 

MS 

F 

A  (Classes) 

162.  89 

1 

162.  89 

2.  60 

B  (Wings  A,  B) 

266.  25 

1 

266. 25 

4.  25* 

AB 

25.  08 

1 

25.08 

0.40 

Within  Cell 

500. 80 

8 

62.  60 

*F  0.90  (1,8)  =  3.46 
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figure  13  appear  related  to  this  change  in  emphasis.  The  first  performance 
of  Nav  Stab,  especially  in  the  trainer  (which  offers  the  best  opportunity  for 
repeated  attempts  to  accurately  stabilize  the  system),  tends  to  be  considerably 
better  than  the  second  performances.  Evidence  of  beneficial  training  effects 
are  provided  by  improved  subsequent  performance  levels  which  may  indicate 
that  the  students  are  learning  to  integrate  this  activity  into  the  mission  con¬ 
text  at  more  adequate  levels  of  performance.  This  conclusion  should  be 
considered  tentative,  however,  in  that  there  was  insufficient  data  to  statistically 
test  the  change  in  WST  performance  levels  and  ASW  performance  changes  were 
not  found  to  be  significant  (see  table  5).  This  lack  of  significance  was  antici¬ 
pated  in  that  ASW  accuracy  values  tend  to  vary  substantially  as  a  function  of 
equipment  operating  status  variations  (e.g.,  calibration  status)  and  the  sample 
size  was  quite  small.  No  comment  can  be  made  on  transfer  of  training  effects 
as  there  was  insufficient  data  to  permit  any  type  of  evaluation. 

SEARCH  PHASE  PROCEDURES,  ACCURACY,  AND  EFFICIENCY.  Changes 
in  student  procedural  performance  of  the  search  phase  are  displayed  in  figure 
14.  The  curves  for  both  WST  and  ASW  indicate  a  trend  to  improve  procedural 
performance  as  a  function  of  performance  repetition.  Statistical  tests  of  the 
performance  level  change  substantiate  the  apparent  learning  effects  in  that  the 
difference  between  the  first  and  second  WST  performance  was  found  significant 
at  p  <  0. 10,  while  the  differences  between  the  first  and  second  ASW  perfor¬ 
mances,  the  classes,  and  the  ASW  x  Class  interaction  were  all  found  significant 
at  p  <  0.01  (see  tables  6  and  7).  The  class  difference  is  interesting  in  that  the 
interaction  effect  appears  to  have  resulted  from  different  initial  performance 
levels  and  similar  terminal  levels,  but  again,  these  results  could  be  explained 
by  a  number  of  factors  including  a  possible  difference  in  prior  training.  What 
is  of  primary  importance  is  that  the  students  did  not  initially  perform  all  the 
search  procedures  correctly,  but  that  they  did  learn  to  do  so  while  in  training. 
Whether  or  not  there  were  any  transfer  of  training  effects  could  not  be  deter¬ 
mined  because  of  insufficient  data. 

It  was  extremely  difficult  to  gather  transfer  data  because  of  training  crew 
configuration.  The  TACCO/Navigator  instructors  were  especially  faced  with 
the  problem  of  often  having  more  than  the  one  TACCO  and  one  Navigator  student 
per  crew.  Their  program  and  schedule  is  designed  for  one  of  each.  As  a  re¬ 
sult,  they  generally  treated  the  training  device  and  the  aircraft  as  equivalent 
training  situations  and  traded  off  the  students  in  position  accordingly.  Thus, 
if  one  TACCO/Navigator  pair  performed  the  first  WST  session,  then  a  second 
TACCO/Navigator  pair  performed  the  first  ASW  session.  For  study  purposes, 
it  could  not  be  said  that  the  second  TACCO/Navigator  pair  had  not  received 
any  training  because  they  had  been  present  as  over -the -shoulder ,  and  some¬ 
times  interactive,  observers  of  performance  by  the  first  team  and  the  instruc¬ 
tion  given  by  the  instructors.  Consequently,  the  only  data  provided  in  these 
situations  was  the  first  performance  by  the  first  team  of  a  crew;  subsequent 
performances  in  the  alternative  training  environment  by  a  second  team  did  not 
satisfy  the  requirements  for  either  experimental  or  control  group  membership. 

Insufficient  da.ta  were  collected  on  the  measure  of  Search  Phase  Accuracy 
to  permit  an  evaluation.  No  trends  were  evidenced  by  the  sample  obtained 
C.’  -  3  for  two  performance  repetitions). 
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TABLE  5.  ANALYSIS  OF  VARIANCE  SUMMARY  FOR  ASW 
TRAINING  EFFECTIVENESS  FOR  NAVIGATION  STABILIZATION  ACCURACY 


Source  of  Variation 

SS 

df 

MS 

F 

Between  Subjects 

4 

A  (Classes) 

5  24.  34 

1 

524. 34 

— 

Subjects  Within  Group 

35223. 25 

3 

11407. 75 

Within  Subjects 

5 

B  (ASW  Nav  Stabs) 

466. 99 

1 

466. 99 

— 

AB 

1702.  64 

1 

1702.  64 

— 

B  x  Subjects  Within  Group 

19146. 59 

3 

6382. 20 

TABLE  6.  ANALYSIS  OF  VARIANCE  SUMMARY  FOR  WST 
TRAINING  EFFECTIVENESS  FOR  SEARCH  PHASE  PROCEDURES 


Source  of  Variation 

SS 

df 

MS 

F 

Between  Subjects 

A  (Classes) 

4.  91 

1 

4.91 

— 

Subjects  Within  Groups 

225. 82 

12 

18.92 

Within  Subjects 

B  (WST  Search  Perfor¬ 
mances) 

155.88 

1 

155. 88 

j 

4. 16* 

AB 

70.  92 

1 

70.92 

1. 89 

B  x  Subjects  Within  Groups 

449.43 

mm 

37.45 

'  F  0.  90  (1,12)  =  3.  18 
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Figure  14.  Team:  Search  Phase  Procedures 


FIRST  CLASS 
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Figure  15.  Class  Differences  in  WST 
Performance  of  Search 
Procedures 


FIRST  CLASS 
—  —  —  (Ns  -  4,  2) 

— -  ....  SECOND  CLASS 


SEARCHES/  NO.  PERFORMED 


Figure  16.  Class  Differences  in  ASW 
Performance  of  Search 
Procedures 
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TABLE  7.  ANALYSIS  OF  VARIANCE  SUMMARY  FOR 
ASW  TRAINING  EFFECTIVENESS  FOR  SEARCH  PHASE  PROCEDURES 


Source  of  Variation 

SS 

— 

df 

MS 

F 

Between  Subjects 

A  (Classes) 

1339.40 

1 

1339.40 

29. 69* 

Subjects  Within  Groups 

270. 67 

6 

45.  11 

Within  Subjects 

B  (ASW  Search 
Performance) 

2316. 62 

1 

2316. 62 

51.35* 

AB 

1339. 37 

1 

1339. 37 

29. 69* 

B  x  Subjects  Within  Groups 

270. 66 

6 

45.  11 

*F  0.99  (1,6)  =  3.7 


The  WST  and  ASW  performance  curves  presented  in  figure  17  indicate 
poorer  efficiency  during  the  first  Search  Phase  performed.  Subsequent  aver¬ 
age  performances  appear  to  vary  within  the  interval  of  155  to  162,  Examina¬ 
tion  of  individual  performances  indicate  that,  indeed,  this  is  the  case.  Many, 
but  not  all,  searches  conducted  by  student  teams  for  the  first  time  required  a 
considerable  amount  of  time. 

Subsequent  performances  improved  until  the  interval  of  155  to  170  was 
reached,  performance  then  varied  within  this  interval  probably  as  a  function 
of  situational  factors  affecting  mission  activities.  It  would  appear,  then,  that 
Search  Phase  efficiency  at  least  benefits  from  initial  training.  Insufficient 
data  were  available  to  statistically  test  this  conclusion. 

The  transfer  of  training  hypotheses  could  not  be  statistically  evaluated 
again  because  of  insufficient  data,  but  it  can  be  said  that  the  available  data 
(Ns  -  1  to  5)  support  both  hypotheses;  that  is,  positive  transfer  of  training 
from  the  device  to  the  aircraft  environment  and  vice  versa. 

LOCALIZATION  PHASE  PROCEDURES,  Curves  for  Localization  Phase 
procedural  performance  appear  in  figure  18.  Neither  these  curves  nor  a 
statistical  test  of  the  WST  curve  nor  inspection  of  the  ASW  data  provide  evi¬ 
dence  of  any  consistent  performance  trends.  The  only  apparent  factor 
determining  performance  variance  was  that  of  class,  again  with  the  second 
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SEARCHES  (NUMBER  PERFORMED) 


Figure  17.  Crew:  Search  Phase  Efficiency 
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WST  (Ns  =  6  TO  13) 
ASW  (Ns  =  4  TO  10) 


LOCALIZATIONS  (NUMBER 
PERFORMED) 


NOTE:  The  WST  curve  is  based  on  independent  samples 
while  the  ASW  curve  is  based  on  incomplete 
repeated  measures. 


Figure  18.  Crew:  Localization  Phase  Procedures 
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class  performing  at  a  higher  level  than  the  first  class  (see  able  8).  Insuffi¬ 
cient  data  were  available  to  analyze  transfer  of  training  effects.  It  can  only 
be  said  that  the  data  did  not  contradict  these  hypotheses. 

TABLE  8.  ANALYSIS  OF  VARIANCE  SUMMARY  FOR  WST 
TRAINING  EFFECTIVENESS  FOR  LOCALIZATION  PHASE  PROCEDURES 


Source  of  Variation 

SS 

df 

MS 

r"  " . . 

F 

Between  Students 

A  (Classes) 

450. 88 

1 

450. 88 

3.  57* 

Students  Within  Group 

1137.  12 

9 

126. 35 

Within  Students 

B  (WST  Localizations 

Performed) 

27.  96 

1 

27.  96 

— 

AB 

0.  00 

1 

0.00 

— 

B  x  Students  Within  Groups 

1359. 49 

| 

9 

1 

j 

151.05 

J 

1 _ 1 

F  0.  90  (1,  9)  =  3.  36 


It  will  be  recalled  that  the  first  study  provided  indications  of  WST  learning 
effects  for  Localization  Phase  accuracy  and  efficiency.  No  analyzabie  data 
were  obtained  on  these  measures  in  the  second  study. 

It  can  only  be  concluded  from  the  above  that  the  effectiveness  of  the  training 
program  and  device  for  Localization  Phase  training  has  not  yet  been  satisfac¬ 
torily  demonstrated  empirically.  Target  localization  can  be  realized  by  com¬ 
plex  and  alternative  activities,  the  performance  of  which  tends  to  be  contingent 
on  situational  characteristics.  Observations  of  crews  in  training  in  the  device 
have  led  to  the  conclusion  that  student  crews  do  learn  the  nature  of  the  localiza¬ 
tion  problem  and  do  tend  to  improve  their  performance  of  localization  activities 
The  inability  to  define  a  meaningful  localization  task  measurement  set  for  the 
two  training  environments  which  would  provide  consistent  repeated  measures 
precluded  adequate  empirical  demonstration  of  the  qualitative  conclusion. 

INDIVIDUAL  PERFORMANCE  SCORES 

NAVIGATOR.  Figure  19  presents  the  mean  WST  and  ASW  performance  curves 
for  the  student  navigators.  The  presence  of  more  than  one  student  navigator 
on  many  teams  resulted  in  students  frequently  either  performing  some  part  of 
each  session  or  sharing  and  trading  off  navigator  functions  during  the  sessions; 
particularly  in  the  WST  environment.  As  a  result,  the  following  two  conditions 
existed:  (1)  it  was  possible  to  use  the  WST  and  ASW  session  number  to  define 
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the  abscissa,  and  (2)  to  ensure  repeated  measures  of  comparable  definition  it 
was  necessary  to  delete  many  of  the  available  scores.  Therefore,  the  perfor¬ 
mance  curves  can  best  be  described  as  representing  independent  groups  be¬ 
cause  of  the  repeated  measure  discontinuities. 

The  ASW  performance  curve  displays  a  consistent  trend  toward  improved 
performance  during  the  training  program.  The  WST  curve  on  the  other  hand 
does  not  appear  to  display  any  consistent  trend.  Since  there  was  a  larger  data 
sample  available  on  this  measure  during  the  entire  training  program,  it  was 
possible  to  develop  individual  curves  per  class  and  per  wing.  This  was  done 
to  evaluate  possible  effects  of  class  and  training  schedule  differences  on  navi¬ 
gator  ASW  and  WST  performance  trends.  (Class  distinctions  were  maintained 
due  to  the  possible  impact  of  differences  in  a  prior  systems  course.  )  These 
trends  are  presented  in  figures  20  and  21  and  may  provide  a  possible  explana¬ 
tion  for  the  curves  in  figure  19.  The  curves  display  one  difference  tendency 
which  had  been  anticipated  from  analysis  of  data  collected  on  other  measures 
and  another  difference  tendency  which  had  been  hypothesized.  That  is,  the 
initial  performance  levels  for  the  second  class  were  higher  in  both  the  WST 
and  ASW  situation,  and  the  wings  receiving  one  or  two  WSTs  prior  to  an  ASW 
flight  tended  to  perform  better  on  both  their  fifth  WST  session,  and  more 
importantly,  their  fifth  ASW  flights. 

Other  variations  between  the  curves  in  figures  20  and  21  were  of  interest 
in  that  they  may  correlate  with  observations  of  student  behavior  in  the  training 
device.  It  was  observed  that  certain  of  the  crews  appeared  to  more  readily 
accept  the  utility  of  sessions  in  the  training  device  than  did  other  crews.  It  is 
suggested  that  those  crews  receiving  their  initial  one  or  two  sessions  in  the 
WST  may  better  accept  the  training  device.  It  can  be  observed  that  these  are 
the  teams  (Wing  A)  which  evidence  more  consistently  patterned  curves,  it  can 
also  be  observed  that  Wing  B  of  the  second  class  provided  the  poorest  average 
WST  5  performances,  where  WST  5  followed  ASW  4  and  5  in  sequence.  Fur¬ 
ther,  Wing  B  of  the  first  class  provided  the  poorest  average  ASW  1  perfor¬ 
mances,  where  ASW  3  followed  WST  2  and  3  in  sequence.  The  evidence  is  in¬ 
conclusive  but  it  does  suggest  that  the  utility  of  a  training  device  may  be  at 
least  partly  a  function  of  when  the  student  is  introduced  to  it. 

The  nature  of  the  analyses  (i.  e.  ,  the  factors  and  factor  levels  evaluated) 
were  a  function  of  which  population  samples  provided  comparable  data  with  the 
necessary  definitions.  The  statistical  analyses  that  could  he  performed  gen¬ 
erally  support  the  trends  displayed  in  figures  20  and  21.  That  is,  the  WST 
performance  change  between  the  second  and  fifth  sessions  was  significant  for 
the  first  class  (p  <  0.05,  table  9).  The  ASW  performance  change  between 
the  first  and  fourth  ASW  flights  were  significant  for  the  first  class  (p  <  0.05, 
table  11)  and,  while  ASW  1  versus  ASW  5  differences  were  not  significant  for 
the  second  class  ( t  able  10),  a  tendency  did  exist  (t  =  1.18,  df-7,  p<0.20). 

Transfer  of  device  training  to  ASW  1  was  tested  and  found  not  significant 
(  table  12).  The  data  tend  to  support  the  transfer  hypothesis  but  interstudent 
differences  were  too  great  to  permit  any  conclusions.  Again,  the  class  factor 
was  found  to  be  a  determiner  of  performance  levels  (p  <  0.05). 
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Figure  20.  Navigator  WST  Performance  as  a  Function  of  Training 
Schedule  and  Class 
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Figure  21.  Navigator  ASW  Performance  as  a  Function  of  Training 
Schedule  and  Class 


5  9 


-1 


NAVTRAEQUIPCEN  70-C-0258-2 

TABLE  9.  ANALYSIS  OF  VARIANCE  SUMMARY  FOR  WST 
TRAINING  EFFECTIVENESS  FOR  NAVIGATOR  STUDENTS  IN  FIRST  CLASS 


Source  of  Variation 

SS 

df 

MS 

F 

Between  Students 

89.8 

2 

Within  Students 

494.  8 

3 

WST  Sessions 

454.  2 

1 

454.  2 

22.  37* 

Residual 

40.  6 

2 

20.  3 

Total 

584.  6 

5 

*F  0.95  (1,2)  =  18.5 

TABLE  10.  ANALYSIS  OF  VARIANCE  SUMMARY  FOR  WST 
TRAINING  EFFECTIVENESS  FOR  NAVIGATOR  STUDENTS  IN  SECONDCLASS 


Source  of  Variance 

SS 

df 

MS 

F 

Between  Subjects 

47.  7 

3 

A  (WST  ) 

11.  8 

1 

11.  8 

0.  66 

Subjects  Within  Group 

35.9 

2 

18.0m 

Within  Subjects 

287.  7 

4 

B  (Wing  ) 

43.  8 

1 

43.  8 

3.  56 

AB 

219.4 

1 

219.  4 

17. 84* 

B  x  Subjects  Within 
Groups 

24.  5 

- 

2 

12.  3 

*F  0.  90  (1,  2)  =  8.  53 
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TABLE  11.  ANALYSIS  OF  VARIANCE  SUMMARY  FOR  ASW 
TRAINING  EFFECTIVENESS  FOR  NAVIGATOR  STUDENT  IN  FIRST  C LASS 


Source  of  Variation 

SS 

df 

MS 

F 

Between  Students 

195.7 

3 

A  (Wings  A,  B) 

131.9 

1 

131.9 

4.13 

Students  Within  Groups 

63.  8 

2 

31.9 

Within  Students 

241. 8 

4 

B  (ASW  1,  4) 

200.0 

1 

200.0 

25. 00* 

AB 

25.9 

1 

25.9 

3.  24 

B  x  Students  Within 
Group 

15.9 

2 

8.  0 

*F0.95  (1,  2)  =  18.  5 

TABLE  12.  ANALYSIS  OF  VARIANCE  SUMMARY  FOR  TRANSFER 
OF  NAVIGATOR  WST  TRAINING  TO  ASW  TRAINING 


Source  of  Variation 

SS 

df 

MS 

F 

A  (Wings  A,  B) 

17.  8 

1 

17.  8 

B  (Class) 

458.  7 

1 

458.  7 

7.  36* 

AB 

7.  8 

1 

7.  8 

Within  Cell 

311. 7 

5 

62.  3 

*F  0.95  (1.5)  =  6.61 
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JEZEBEL  OPERATOR  ANNOTATION  ACCURACY  AND  COMPLETENESS.  The 
Jezebel  operator  performance  curves  (  figure  22)  display  a  consistent  learning 
trend  up  to  the  fourth  session  (most  Jezebel  students  performed  a  sufficient 
part  of  each  session  to  permit  scoring;  as  a  result  it  was  possible  to  use  the 
WST  and  ASW  session  number  as  an  abscissa  definition).  The  results  of  statis¬ 
tical  tests  of  these  trends  were  not  significant  for  either  WST  or  ASW  training 
because  of  the  small  sample  size  (Ns  =  3,4)  and  inter  student  differences  (see 
tables  9  and  10).  The  data,  however,  consistently  indicate  performance  im¬ 
provement  in  both  the  WST  and  ASW  environments  with  a  ceiling  on  WST 
performance.  * 

Transfer  of  training  from  WST  to  ASW  is  supported  by  the  available  data 
(mean  scores  of  84%  versus  78%  based  on  Ns  of  4  and  3),  but  the  conclusion 
is  again  tentative  because  of  small  sample  sizes.  Insufficient  data  were  ob¬ 
tained  to  permit  evaluation  of  transfer  of  training  from  ASW  to  WST. 

RADAR /MAD  OPERATOR.  Insufficient  data  were  obtained  to  perform  any 
statistical  analyses  of  learning  trends  or  to  perform  any  type  of  transfer  of 
training  evaluation.  Average  performance  levels  as  a  function  of  WST  and 
ASW  session  number  are  depicted  in  Figures  23,  24,  and  25.  These  curves 
will  be  the  basis  for  evaluation  of  learning  trends. 

RADAR  LOG  ENTRIES.  Both  the  WST  and  ASW  curves  (figure  23)  depict  per¬ 
formance  improvement  with  perfect  terminal  scores  (110%)  by  all  three 
students.  It  appears  this  task  is  well  learned  during  the  normal  program  of 
training. 

MAD  PROCEDURES.  The  curves  presented  in  figure  24  do  not  display  any 
consistent  trends.  Whether  or  not  a  larger  sample  of  students  would  have  dis¬ 
played  the  same  inconsistent  pattern  cannot  be  determined  from  available  infor¬ 
mation.  It  can  only  be  said  that  the  mission  context  for  MAD  procedures  and 
the  poor  display  simulation  in  both  training  environments  (training  device  and 
aircraft)  may  preclude  effective  learning  to  a  consistent  100%  level  of  perfor¬ 
mance  within  this  training  program. 

MAD  TAPE  ANNOTATIONS.  These  performance  curves  (figure  25)  do  display 
relatively  consistent  trends  toward  improved  performance  and,  based  on  this 
evidence,  it  appears  that  learning  has  occurred. 


It  is  hypothesized  that  this  WST  ceiling  effect  is  a  function  of  the  poor  simu¬ 
lation  of  Jezebel  and  DIFAR  signal  characteristics  inherent  in  rhe  training 
device  at  the  time  when  this  study  was  conducted.  Since  annotation  activity 
tends  to  reflect  the  students  interpretation  of  signal  characteristics,  it  is 
possible  that  training  students  to  a  100%  annotation  criterion  in  the  device 
would  result  in  initial  negative  transfer  effects  in  the  ASW  environment. 
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TABLE  1  L  ANALYSIS  OF  VARIANCE  SUMMARY  FOR  WST  TRAINING 
EFFECTIVENESS  FOR  JEZEBEL  OPERATOR  ANNOTATION 

PERFORMANCE 


Source  of  Variation 

SS 

df 

MS 

F 

Between  Students 

507.  37 

3 

Within  Students 

1049.50 

4 

WST  Sessions 

630.12 

1 

630. 12 

4.  51* 

Residual 

419.38 

3 

139. 79 

Total 

1556. 87 

7 

139. 79 

' Not  significant  at  the  0.10  level  (F  0.75  (1,3)  =  2.C2) 


TABLE  14.  ANALYSIS  OF  VARIANCE  SUMMARY  FOR  ASW 
TRAINING  EFFECTIVENESS  FOR  JEZEBEL 
OPERATOR  ANNOTATION  PERFORMANCE 


Source  of  Variation 

SS 

df 

MS 

F 

Between  Students 

416.33 

2 

Within  Students 

685.00 

3 

ASW  Flights 

450.66 

1 

450. 66 

3.85* 

Residual 

234. 34 

2 

117.  17 

Total 

1101.33 

5 

Not  significant  at  the  0.  10  level  (F  0.  75  (1,  2)  =  2.  57) 
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RADAR /MAD  OPERATOR 

RADAR:  PROCEDURES  (ALL  STUDENTS  RETAINED  IN  PROGRAM 
SCORED  100%  FROM  BEGINNING 

RADAR:  LOG  ANNOTATIONS,  %  COMPLETE 
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Figure  23.  Individual:  Radar /MAD  Operator  Radar  Log  Entries 
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RADAR /MAD  OPERATOR 

MAD:  PERCENT  OF  PROCEDURES  CORRECTLY 
PERFORMED  AS  REQUIRED 


WST 

ASW 


(Ns  =  2  TO  3) 
(Ns  =  2  TO  3) 


Figure  24.  Individual:  Radar/MAD  Operator  Performance 
of  MAD  Procedures 

MAD:  LOG  ANNOTATIONS,  PERCENT  COMPLETE 
AND  ACCURATE  (Ns  =  2  TO  3) 


WST  (Ns  =  2  TO  3) 
ASW  (Ns  =  2  TO  3) 


'igure  25.  Individual:  Radar/MAD  Operator  MAD  Tape  Annotations 
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SECTION  IV 

QUESTIONNAIRE  EVALUATION  OF  DEVICE  2F69B 


INTRODUCTION 

Instructor  personnel  completed  a  questionnaire  requiring  ratings  and  com¬ 
ments  regarding  the  P3A/B  training  device  as  an  instructional  tool  for  their 
individual  positions  (Appendix  D).  Questionnaire  items  relate  to  device  char¬ 
acteristics  and  utilities  in  terms  of  task  e.  'ironment  realism,  task  perfor¬ 
mance  realism,  and  training  effectiveness.  Because  these  personnel  actually 
use  two  simulations  of  the  real  world  as  instructional  tools,  the  training  de¬ 
vice  and  aircraft,  certain  questions  required  judgments  regarding  both  tools. 
Further,  the  judged  value  of  inflight  training  missions  and,  therfore,  the  re¬ 
lative  value  of  the  training  device,  was  anticipated  to  be  a  function  of  the  tar¬ 
gets  available  (surface  vessels  versus  submarines).  Therefore,  questionnaire 
items  requiring  evaluations  of  the  training  device  in  comparison  to  the  training 
aircraft  suggested  that  the  respondee  provide  separate  ratings  .f  he  felt  the 
type  of  target  available  altered  the  value  of  inflight  training  missions. 

Instructor  ratings  of  device  characteristics  are  presented  in  the  initial  par¬ 
agraphs  below.  These  presentations  are  followed  by  discussions  of  instructor 
comments . 

DEVICE  CHARACTERISTICS  RATINGS 

DEVICE  REALISM.  Realism  was  evaluated  from  three  standpoints  (1)  over¬ 
all  realism  of  the  device  and  aircraft  training  environments  (  table  15),  (2)  de¬ 
vice  task  environment  realism  as  compared  to  the  inflight  situation  (  fcigure  26 
and  table  16),  and  (3)  task  performance  realism  in  the  device  as  compared  to 
inflight  task  performance  (  figure  27  and  table  17). 

It  can  be  seen  from  (table  15  that  the  aircraft  is  considered  to  present  the 
more  realistic  environment  for  the  pilot,  navigator,  and  radar/MAD  operator. 
The  trainer,  on  the  other  hand,  was  considered  to  be  more  realistic  for  the 
TACCO  and  Jezebel  operator.  This  latter  situation  may  be  explained  by  the 
fact  that  ASW  training  flights  must  usually  use  surface  vessels  as  targets. 
These  vessels  do  not  provide  signals  with  appropriate  characteristics  to  the 
Jezebel  operator  and  will  not  simulate  target  behaviors  in  response  to  P3A/B 
actions.  This  contention  is  supported  by  tables  16  and  17,  which  describe 
evaluated  device  realism  as  compared  to  flights  using  surface  vessels  versus 
those  using  submarines.  The  device  is  considered  comparatively  realistic  in 
the  first  instance,  but  considerably  less  so  in  the  second  instance. 

The  relatively  low  values  given  both  training  environments  for  the  Jezebel 
operator  position  is  apparently  due  not  only  to  the  low  fidelity  simulation  of 
target  characteristics  in  both  cases,  but  also  to  the  fact  that  many  Jezebel 
students  were  training  for  assignments  to  DIFAR  retrofit  aircraft.  As  there 
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TABLE  15.  AVERAGE  INSTRUCTOR  EVALUATIONS  OF  THE  REALISM 
OF  TRAINING  DEVICE  AND  AIRCRAFT  TRAINING  ENVIRONMENT 


Training  Environment 
Realism 

Position 

Trainer 

Aircraft 

Pilot/Copiiot  (N  =  2) 

2.4* 

4.4 

TACCO/Navigator  (N  =  9) 

4.  0 

3.2 

TACCO  (N  =  S) 

4.3 

3.3 

Navigator  (N  =  S) 

3.3 

3.9 

Jezebel  Operator  (N  =  7) 

2.7 

1.9 

Radar/MAD  Operator  (N  =  S) 

Lll_i 

4.  0 

_ 

A  five  point  continuum  was  provided  for  each  questionnaire  item  requiring  a 
rating  response.  Scale  responses  were  quantified  using  a  1 . 0  to  5.0  value 
range. 
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Figure  26.  Trainer  Evaluation:  Task  Environment  Realism 
(N  =  2,  9,  7,  5) 
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TABLE  16.  COMPARISON  OF  TRAINER  TASK  ENVIRONMENT  REALISMS 
WHEN  EVALUATED  AGAINST  SURFACE  VESSEL  TRAINING 
FLIGHTS  VERSUS  SUBMARINE  TRAINING  FLIGHTS 


Task  Environment  Factors/ 
Positions 

Trainer  Task  Environment  Realism 

Compared  to 
Surface  Vessel 
Training  Flights 

Compared  to 
Submarine  Training 
Flights 

Target  Information 

Pilot/Copilot  (N  =  1 ) 

5.0 

4.  0 

TACCO /Navigator  (N  =  5) 

4.  2 

2.2 

Jezebel  (N  -  3 ) 

4.2  x  =  4.  3 

1.8  x  =  24 

Radar  /MAD  (N  2) 

1.8 

1 . 6 

Buoy  Information 

TACCO/Navigato r  (N  =  4) 

4. 0 

2.3 

Jezebel  (N  ^  2) 

4.2  x  -  4.  1 

2.0  x  =  2.2 

Electronic/Real  World  Match 

TACCO/Navigator  (N  =  2) 

3.5 

3.  3 

Jezebel  (N  -  3) 

1 _ 

3.3  "  -  3.4 

2.8  x  =  3.  1 
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Figure  27.  Trainer  Evaluations:  Task  Performance  Realism  (N  2,  8,  6,  4) 
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TABLE  17.  COMPARISON  OF  TASK  PERFORMANCE  REALISMS  IN  THE 
TRAINER  WHEN  EVALUATED  AGAINST  SURFACE  VESSEL  TRAINING 
FLIGHTS  VERSUS  SUBMARINE  TRAINING  FLIGHTS 


Trainer  Task  Performance  Realism 

Tasks  Performed/Positions 

Compared  to 
Surface  Vessel 
Training  Flights 

Compared  to 
Submarine  Training 
Flights 

TACCO/Sensor  Operator  Interaction 

TAC  CO /Navigator 

4.  3 

2.  7 

Jezebel 

3.9 

3.2 

Radar/MAD 

3.8 

1 . 6 

TACCO /Pilot  Guidance  Control 

TACCO/  Navigator 

2.  5 

1.  5 

TACCO/Navigator  Target  Course 
Estimation 

3.  5 

2.  5 

TACCO/Navigator  Target  Speed 
Estimation 

3.5 

2.  5 

TACCO/Navigator  Target  Position 
Estimation 

3.  5 

2.  5 
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are  design  and  capability  differences  between  theDIFAR  station  in  retrofitted 
aircraft  and  the  Jezebel  station  in  standard  P3A/B  aircraft,  both  the  trainer 
and  the  aircraft  presented  low  fidelity  simulations  for  these  students. 

Task  environment  realism  with  respect  to  information  presentations  gen¬ 
erally  tended  to  be  rated  moderately  high  for  the  pilot  and  TACCO  positions 
but  less  so  for  the  navigator,  Jezebel,  and  radar/MAD  positions  (  figure  26). 
The  unweighted  means  for  environmental  realism  characteristics  range  from 
2.8  to  3.4.  In  contrast,  task  performance  realism,  in  terms  of  crew  mem¬ 
ber  interactions,  generally  received  higher  ratings  (  figure  27).  These  means 
ranged  from  3.5  to  4.1. 

DEVICE  EFFECTIVENESS.  Effectiveness  was  also  evaluated  from  three 
standpoints:  (1)  general  effectiveness  characteristics  (  figure  28  and  table 
18),  (2)  learning  objectives  (  figure  29  and  table  19),  and  (3)  alternative  learn 
ing  phases  (tables  20  and  21).  tables  18,  19,  and  21  represent  the  subset  of 
TACCO/Navigator  instructor  personnel  who  provided  separate  responses  for 
these  two  positions. 

The  deivce  was  rated  as  fairly  effective  overall  and  as  compared  to  inflight 
training  (figure  28  and  table  18)  for  all  positions  except  the  pilot  and  Naviga¬ 
tor.  In  contrast,  the  problems  and  problem  characteristics  were  rated  as 
quite  effective  for  only  the  TACCO  and  Navigator. 

The  training  device  was  rated  as  particularly  effective  for  the  learning  ob¬ 
jectives  of  knobology,  procedural  use  of  equipment,  utilization  of  sensor  infor 
mation,  and  especially,  team  coordination  (figure  29).  It  was  given  compar¬ 
atively  low  ratings  for  effectiveness  in  training  sensor  data  interpretation  at 
the  Jezebel  and  Radar/MAD  positions,  while  the  TACCO  position  received  a 
higher  rating.  These  poorer  ratings  appear  to  be  due  to  Low  fidelity  sensor 
data  inputs  to  these  stations. 

The  effectiveness  of  the  device  and  the  aircraft  was  also  found  to  vary  per 
position  as  a  function  of  training  phase  (  tables  20  and  21),  The  comparative 
effectiveness  of  the  two  training  environments  per  position,  across  training 
phases,  is  shown  by  the  unweighted  position  means  in  these  two  tables.  The 
device  is  rated  as  best  overall  for  TACCO  students,  while  the  inflight  aircraft 
is  rated  as  best  overall  for  the  pilot,  navigator,  and  Radar/MAD  students. 

The  environments  received  equivalent  ratings  for  the  Jezebel  operator. 

Of  particular  interest  is  the  comparison  of  the  overall  rated  effectiveness 
of  these  two  training  environments  per  training  phase  (table  22).  The  trainer 
and  aircraft  were  evaluated  as  roughly  equivalent  for  the  first  two  phases; 
Demonstration  and  Practice,  and  Development  of  Proficiency.  The  aircraft 
received  a  decidedly  higher  average  rating  for  the  Refinement  of  Technique 
phase.  For  Development  of  New  Tactics,  however,  the  trainer  was  evalu¬ 
ated  as  equal  or  better  than  the  aircraft. 
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Figure  28.  Trainer  Evaluations:  General  Training 
Effectiveness  (N  =  3,  9,  7,  5) 
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TABLE  18.  COMPARISON  OF  GENERAL  TRAINING  EFFECTIVENESS 
EVALUATIONS  FOR  THE  TACCO  AND  NAVIGATOR  POSITIONS  (N  =  5) 


General  Training 
Effectiveness 

Effectiveness  Characteristics 

TACCO 

Navigator 

Overall  Effectiveness 

3.8 

3.  1 

Problems,  Problem  Characteristics 

3.8 

3.7 

Compared  to  Inflight  Training 

3.5 

2.8 
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Figure  29.  Trainer  Evaluations:  Learning  Objective  Training 
Effectiveness  (N  -  2,  9,  7,  5)  (Part  1  of  2) 


NAVTRAEQUIPCEN  70-C-025P-2 


UNWEIGin  El) 
MEAN 


TACTICS 


J.  7 


A 


TEAM 

COORDINAT  [ON 


4.4 


A - At 

LOW 

EFFECTIVENESS 


4- 


Ar - - A 

HIGH 

EFFECTIVEN  ESS 


PILOT/COPILOT  INSTRUCTORS 


JEZEBEL  INSTRUCTORS 


TACCO/NAVIGATOR  INSTRUCTORS  fjjjljjiilijjli 


RADAR/ MAD  INSTRUCTORS 


Figure  29.  Trainer  Evaluations:  Learning  Objective 
Training  Effectiveness  (N  =  2,  9,  7,  5) 
(Part  2  of  2) 
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TABLE  19.  COMPARISON  OF  LEARNING  OBJECTIVE  TRAINING 
EFFECTIVENESS  EVALUATIONS  FOR  THE  TACCO  AND 
NAVIGATOR  POSITIONS  IN  THE  TRAINER 


Training  Effectiveness 

Learning  Objectives 

TACCO 

Navigator 

Knobology  (N  =  4) 

4.6 

3.4 

Procedural  Use  of  Equip¬ 
ment  (N  =  4) 

4.  6 

3.6 

Utilization  of  Sensor  Infor¬ 
mation  (N  =  2) 

4.2 

4.2 

Methodology  and  Technique 
Development  (N  =  3) 

3.9 

3.8 

Tactics  (N  =  2) 

4.  1 

3.8 

Team  Coordination  (N  =  3) 

4.3 

4.  2 

■>6 
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TABLE  20.  AVERAGE  INSTRUCTOR  EVALUATIONS  OF  THE  EFFECTIVE 
NESS  OF  TRAINING  DEVICE  AND  AIRCRAFT  TRAINING  ENVIRONMENTS 
FOR  ALTERNATIVE  TRAINING  PHASES 


Position/Training  Phase 

Training  Environment 
Effectiveness 

Trainer 

Aircraft 

Pilot/Copilot  (N  =  3) 

Demonstration  and  Practice 

2.8  Unweighted 

4.2  Unweighted 

Development  of  Proficiency 

2.8  x  =  3 .  1 

4.2  x  =  4.  0 

Refinement  of  Technique 

3.  1 

3.8 

Development  of  New  Tactics 

3.  5 

3.7 

TACCO/Navigator  (N  =  9) 

Demonstration  and  Practice 

4.4  Unweighted 

3.3  Unweighted 

Development  of  Proficiency 

4.3  x  =  4. 1 

3.3  x  =  3.4 

Refinement  of  Technique 

3.  7 

3.8 

Development  of  New  Tactics 

3.  9 

3.1 

Jezebel  Operator  (N  =  1) 

Demonstration  and  Practice 

3.3  Unweighted 

3.5  Unweighted 

Development  of  Proficiency 

3.3  x  =  3 .  1 

3.4  x  =  3 . 2 

Refinement  of  Technique 

2.4 

3.8 

Development  of  New  Tactics 

3.  5 

1.9 

Radar/MAD  Operator  (N  -  5) 

Demonstration  and  Practice 

3.4  Unweighted 

3.9  Unweighted 

Development  of  Proficiency 

2.7  x  =  3.  0 

3.4  x  =  3.8 

Refinement  of  Technique 

2.8 

4.  0 

Development  of  New  Tactics 

3.  1 

2.7 
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TABLE  21.  COMPARISON  OF  TRAINING  DEVICE  AND  AIRCRAFT  TRAINING 
EFFECTIVENESS  EVALUATION  FOR  THE  TACCO  AND  NAVIGATOR 
POSITIONS  DURING  ALTERNATIVE  TRAINING  PHASES  (N  =  3) 


Training  Phase 

Training  Environment  Effectiveness 

Trainer 

Aircraft 

TACCO 

Navigator 

TACCO 

Navigator 

Demonstration  and  Practice 

4.4 

3.6 

3.1 

4.  3 

Development  of  Proficiency 

4.4 

3.2 

3.0 

4.  1 

Refinement  of  Technique 

4.  1 

2.  9 

3.4 

4.  5 

Development  of  New  Tactics 

4.  3 

3.2 

3.1 

3.9 

Mean  Evaluation 

4.  3 

3.2 

3.2 

4.2 
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TABLE  22.  OVERALL  EFFECTIVENESS  OF  TRAINING  ENVIRONMENTS 
PER  TRAINING  PHASE  (UNWEIGHTED  MEANS) 


Training  Phase 

Training  Environment 

Trainer 

Aircraft 

Demonstration  and  Practice 

3.5 

3.  7 

Development  of  Proficiency 

3.3 

3.6 

Refinement  of  Technique 

3.0 

3.9 

Development  of  New  Tactics 

3.5 

3 .  1  * 

If  Jezebel  operator  excluded,  x  =  3.5 
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INSTRUCTOR  COMMENTS 

Instructor  comments  and  suggestions  to  questionnaire  items  are  presented 
in  the  following  sections. 

PROBLEMS  AND  PROBLEM  CHARACTERISTICS.  Questionnaire  item  2 
queried,  "Would  you  change  the  problem  characteristics  in  any  way  for  train¬ 
ing  purposes?  If  so,  what  changes  would  you  make?"  Instructor  responses 
are  presented  by  position. 

a.  Pilot /Copilot.  Given  the  limited  cockpit  simulation  in  the  WST  trainer 
and  the  inability  to  run  the  WST  and  OFT  trainers  in  a  coupled  mode 
(the  OFT  trainer  is  unavailable  due  to  its  use  as  a  cockpit  trainer), 

no  changes  were  recommended  for  this  position. 

b.  TA CCO /Navigator .  Two  problem  areas  were  pointed  to  which  both 
related  to  the  TACCO  position  and  impacted  on  other  positions.  The 
device  permits  instructors  to  vary  several  important  environmental 
parameters.  The  effective  utilization  of  these  capabilities  are 
limited,  however,  by  two  factors:  (1)  the  several  members  of  any 
one  scudent  team  will  receive  assignments  to  diffeient  fleet  squad¬ 
rons,  which  operate  under  differing  environmental  conditions  and 
constraints;  and  (2)  instructor  personnel  do  not  presently  receive 
formal  training  in  device  capabilities,  operating  characteristics  and 
utilization  for  training  purposes.  These  problem  areas  could  prob¬ 
ably  be  resolved  by  modification  of  fleet  assignment  procedures, 
student  flow  patterns,  and  development  of  a  training  course  for  in¬ 
structor  personnel  in  device  utilization. 

c.  Jezebel  Operator.  Suggestions  made  for  this  position  point  to  two 
problemareas:  (1)  again,  inadequate  instructor  training  with  respect 
to  device  capabilities  and  utilization,  and  (2)  device  limitations.  It 
was  suggested  that  a  greater  variety  of  targets  be  made  available  , 
thus  providing  the  differing  signal  characteristics  required  for  oper¬ 
ator  training,  that  hyperbolic  LOFAR  fix  (HLF)  capabilities  be  im¬ 
proved,  and  that  comparative  LOFAR  listening  (CPA)  capabilities  be 
provided. 

d.  Radar/MAD  Operator.  The  present  training  syllabus  tends  to  be 
mission,  rather  than  crew  oriented.  As  a  result,  the  Radar/MAD 
student  received  irregular  MAD  training  and  very  little  radar  train¬ 
ing.  It  was  strongly  recommended  that  the  student  receive  more 
radar  training,  both  in  the  device  and  in  flight. 

DEVICE  TRAINING  EFFECTIVENESS.  The  first  question  posed  under  item  3 
was:  "Are  there  any  aspects  of  the  ASW  mission  (e.g.  ,  coordination,  target 
information,  etc.  )  for  which  the  trainer  provides  better  training  than  the  in¬ 
flight  training  missions  do?  If  so,  please  discuss  them."  Instructors  answered 
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the  question  asked  and/or  indicated  why  the  trainer  was  a  better  instructional 
tool.  The  instructors  were  unanimous,  across  position,  in  describing  the  whys: 
(1)  the  ability  to  "freeze"  the  problem  during  the  WST  mission  so  that  errors 
can  be  discussed,  proper  procedures  demonstrated,  or  alternative  courses  of 
action  discussed;  (2)  the  ability  to  have  a  submerged  target  and,  further,  to 
control  both  target  and  merchant  vessels  such  that  the  tactical  mission  evolves 
realistically  and  situations  can  be  presented  for  instructional  purposes;  and 
(3)  economies  resulting  from  no  fuel  costs,  no  need  to  aborc  missions  due  to 
downed  equipments,  and  no  time  spent  in  preflight  and  transit  to  an  operations 
area.  Other  responses  are  presented  per  position. 

a.  Pilot/Copilot.  The  device  was  described  a;  "excellent"  for  crew 
coordination  training.  One  instructor  also  commented  that  the  trainer 
was  very  good  for  beginning  ASW  pilots,  while  either  the  trainer  or 
the  aircraft  was  acceptable  for  experienced  ASW  pilots. 

b.  TACCO /Navigator.  The  device  is  viewed  as  being  most  effective  m 
teaching  crew  coordination,  knoboloby,  procedures,  and  localization 
tactics . 

c.  Jezebel  Operator.  All  responses  provided  further  emphasis  for 
(1 )  and  (2)  above. 

d.  Radar /MAD  Operator.  The  five  responses  to  this  question  expressed 
divergent  opinions.  Two  instructors  indicated  device  training  was  not 
a  better  instructional  tool  lor  any  aspect  of  the  operational  ASW  mis¬ 
sion.  One  respondee  agreed,  but  excepted  crew  coordination  training. 
The  other  two  respondees  indicated  the  trainer  was  more  realistic  for 
radar  operations  in  terms  of  target  behavior  and  indicated  that  item 
(1)  above  was  an  advantage. 

AIRCRAFT  TRAINING  EFFECTIVENESS.  The  second  question  posed  under 
questionnaire  item  3  was:  "Are  there  any  aspects  of  the  operational  ASW  mis¬ 
sion  for  which  the  inflight  training  missions  provide  better  training  than  the 
trainer  does?  If  so,  please  discuss  them."  The  instructors  rather  unani¬ 
mously  pointed  to  those  environmental  factors  which  are  not  simulated  in  the 
device.  The  ASW  crew  must  operate  within  the  environmental  constraints  im¬ 
posed  by  the  aircraft  (e.  g.  ,  noise,  vibration,  g-forces,  etc.),  the  airspace 
(e.g.,  wind  and  pressure  variation.-),  and  the  sea  (e.g.,  temperature  levels, 
sea  state).  The  aircraft  is  thus  seen  as  training  the  students  to  operate  with¬ 
in  environmental  constraints  and  to  appropriately  respond  to  environmental 
parameters . 

a.  Pilot  and  Navigator.  External  visual  references  (e.g.,  buoys, 

smoke  bombs)  and  LORAN  an  be  used  in  the  aircraft  but  not  in  the 
simulator.  Thus,  che  aircraft  provides  training  to  these  positions 
which  cannot  presently  be  obtained  in  the  trainer. 
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b.  TACCO.  No  additional  comments. 

c.  Jezebel  Operator.  CPA  patterns  are  not  presented  and  thus  CPA 
activities  can  be  trained  only  in  the  aircraft. 

d.  Radar/MAD  Operator.  Both  the  radar  and  MAD  displays  are  more 
realistic  in  the  aircraft,  thus  providing  better  training  in  sensor 
daia  interpretation.  It  was  pointed  out,  however,  that  the  present 
inflight  training  syllabus  provides  little  opportunity  for  the  radar 
operator  to  perform  and,  therefore,  to  learn  his  tasks  (the  same 
is  true,  incidently,  of  the  WST  syllabus). 

DISPLAYS.  Question  4  required  a  rating  in  response  to,  "How  realistic  are 
the  displays  of  sensor  data  and/or  other  information  at  this  position  in  the 
trainer?",  and  then  asked,  "What  improvements  would  you  suggest,  where?". 

a.  Pilot  /Copilot.  Display  realism  for  this  station  was  generally  de¬ 
scribed  as  "excellent.  "  The  only  exceptions  made  related  to  the 
OTPI  in  that  the  one  in  the  trainer  is  far  more  accurate  than  the 
ones  in  aircraft. 

b.  TACCO /Navi gator.  The  displays  for  these  positions  were  described 
as  ".  .  .  probably  more  realistic  than  any  other  in  the  trainer"  and 
most  of  the  recommendations  given  related  to  the  sensor  stations. 
Recommendations  for  the  TACCO /Navigator  positions  included  up¬ 
dating  the  torpedo  panels  to  include  MK-46s  and  the  additions  of  a 
LORAN  simulation  and  the  ASA-47  D/AM  computer  to  the  Navigator 
station. 

c.  Jezebel  Operator.  Several  instructors  made  the  following  recom¬ 
mendations:  (1)  develop  new  high  fidelity  target  sound  tracks  so 
that  aural  listening  tasks  can  be  learned  in  the  trainer,  (2)  develop 
new  high-fidelity  Jezebel  tapes  so  that  the  student  can  be  exposed  to 
a  wider  selection  of  submarine  types  and  gram  interpretation  skills 
can  be  trained,  (3)  improve  trainer  capability  to  realistically  simu¬ 
late  signal  characteristics  associated  with  HLF  ard  CLF  functions, 
and  (4)  introduce  a  realistic  simulation  of  CPA.  The  low-fidelity 
signal  characteristic  simulations  for  a  limited  inventory  of  out-of- 
date  targets  limits  the  amount  of  training  these  instructors  can  pro¬ 
vide  to  their  Jezebel  students.  A  final  recommendation  was  that  the 
trainer's  AQA-5  be  replaced  with  the  AQA-7. 

d.  Radar/MAD  Operator.  Both  the  radar  and  MAD  displays  were  de¬ 
scribed  as  presenting  very  unrealistic  signal  characteristics,  thus 
limiting  the  amount  of  training  that  can  be  provided.  It  was  recom¬ 
mended  that  both  displays  be  made  more  realistic  overall  (e,  g.  , 
display  sea  states  on  the  radar,  improve  MAD  response  behaviors) 
and  display  signals  with  higher  fidelity. 
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DEVICE  DESIGN.  Question  8  queried,  "In  terms  of  working  with  the  trainee 
during  a  training  session,  which  aspects  of  the  trainer's  design  do  you  feel 
aid  or  hinder  your  efforts  as  an  instructor?"  Many  responses  were  similar 
to  ones  already  made  to  other  questions  (e,g.,  ability  to  "freeze"  the  problem, 
unrealistic  signal  characteristics).  Several  instructors  pointed,  however,  to 
the  ability  to  observe  the  students,  obtain  problem  parameter  readouts  in  the 
instructor  station,  observe  the  optical  mission  plotter,  and  to  control  many 
prob’em  parameters  as  being  very  desirable  design  features.  The  compact¬ 
ness  of  the  design,  causing  instructor  personnel  crowding,  was  pointed  to  as  a 
negative  design  feature.  Position-specific  comments  follow: 

a.  Pilot/Copilot.  The  WST  has  been  designed  to  permit  ordnance  sys¬ 
tem  training  (weapons  employment,  jettison  demonstrations).  Fav¬ 
orable  comments  were  made  regarding  this  design  feature. 

b.  TACCO /Navigator.  Favorable  comments  were  made  regarding  the 
extent  to  which  activities  could  be  monitored,  but  suggestions  were 
made  to  further  improve  this  capability.  It  was  suggested  that  the 
area  directly  behind  these  positions  might  be  elevated  or  otherwise 
altered  so  as  to  permit  better  observation  of  the  DRT  without  inter¬ 
ference.  It  was  also  suggested  that  the  ability  to  monitor  non-ICS 
communications  from  the  instructor's  station  would  permit  better 
evaluation  of  crew  interactions,  especially  between  the  TACCO  and 
Navi  gator. 

c.  Jezebel  Operator.  No  comments  were  made  over  and  above  those 
discussed  above. 

d.  Radar/MAD  Operator.  The  limitation  in  number  of  radar  targets 
was  described  as  a  poor  design  feature.  All  other  comments  have, 
been  discussed  elsewhere. 

STUDENT  PERFORMANCE  EVALUATION.  Question  9  asked,  "Are  there  any 
modifications  or  improvements  you  would  suggest  that  would  allow  better  or 
more  efficient  evaluation  of  student  performance  at  this  station  (i .  e . ,  either 
in  the  form  of  recording  devices,  better  or  more  slave  equipment,  etc.  )?  " 

a.  Pilot/ Copilot.  It  was  recommended  that,  if  the  WS  I  and  OF1  trainers 
cannot  be  used  in  a  coupled  mode,  that  the  pilot  station  be  separated 
from  the  instructor' s  station  and  optical  mission  plotter.  Although 
the  possession  of  special  "inside"  knowledge  by  student  pilots  regard¬ 
ing  mission  progression  did  not  often  appear  to  affect  team  behaviors 
and  performances,  there  were  instances  of  either  positive  effects 
(e.  g. ,  TACCO  prompting  by  the  pilot)  and  negative  effects  (e.  g. , 
pilot  responses  when  he  knew  the  TACCO' s  commands  were  inappro¬ 
priate).  This  feature  of  simulator  design  should  be  eliminated  from 
future  models  of  this  equipment. 
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b.  TACCO /Navigator.  Grading  criteria  and  number  of  students  per 
position  per  crew  were  presented  as  problem  areas.  The  training 
squadron  continually  works  on  upgrading  scoring  criteria  and  devel¬ 
oped  some  new  insights  as  a  result  of  this  study.  The  number  of 
students  in  an  ASW  class  has  been  a  continuing  problem  and  class 
overloads  detract  from  the  instructor's  ability  to  provide  adequate 
training  and  performance  evaluation. 

It  was  also  suggested  that  these  positions  might  benefit  from  print¬ 
outs  of  the  mission  plot  for  later  analysis. 

c.  Jezebel  Operator.  All  responses  to  this  question  have  been  dis¬ 
cussed  elsewhere. 

d.  Radar/MAD  Operator.  No  changes  were  recommended. 

DEVICE  MAINTENANCE.  Question  10  queried,  "Are  there  any  changes  in 
maintenance  you  would  suggest?"  Three  comments  were  made  in  response  to 
this  question  by  instructors  at  all  positions:  (1)  Give  the  trainer  a  higher 
priority  and  a  larger  maintenance  budget  such  that  replacement  parts  will  be 
made  quickly  available  when  needed;  (2)  Provide  a  second  trainer  so  that  ade¬ 
quate  maintenance  downtime  can  be  increased.  At  present,  the  trainer  is  uti¬ 
lized  by  both  training  and  fleet  squadrons.  As  a  result,  the  downtime  for 
maintenance  has  been  reduced  to  an  unacceptable  minimum;  and  (3)  The  main¬ 
tenance  personnel  were  given  high  marks  for  making  the  best  of  a  difficult  situ¬ 
ation.  It  was  also  recommended  that  additional  qualified  personnel  be  added  to 
the  maintenance  staff. 

TRAINING  SCHEDULES.  Question  1 1  asked,  'Would  you  change  the  amount  of 
training  time  presently  spent  in  the  trainer  or  in  the  aircraft,  or  would  you 
suggest  any  different  utilization  of  the  two  training  environments  (c.g.  ,  sched¬ 
uling)?  If  so,  please  discuss  the  changes  you  would  make  and  explain  the  ex¬ 
pected  advantages.  " 

The  consensus  of  opinion  appears  to  be  that:  (1)  the  number  of  ASW  flights 
could  be  decreased  by  at  least  one;  (2)  the  number  of  WST  sessions  could  be 
increased  for  the  TAC-CO,  Navigator,  and  Jezebel  positions;  and  (3)  the  num¬ 
ber  of  both  WST  and  ASW  training  missions  could  be  reduced  for  the  Radar/ 
MAD  position.  Specific  comments  are  presented  below. 

a.  Pilot/Copilot.  Three  instructors  recommended  an  additional  WST 
session  prior  to  the  first  ASW  flight  for  both  the  pilot  and  TACCO 
students.  Only  one  instructor  recommended  no  changes.  In  addi¬ 
tion,  it  was  suggested  that  these  students  would  benefit  by  gaining 
some  experience  in  performing  the  tasks  associated  with  the  sensor 
operator  positions.  It  was  also  suggested  that  all  students  should 
be  allowed  to  observe  instructors  running  through  some  sample 
problems  for  one  WST  session,  both  for  instructional  purposes  and 
to  provide  the  students  with  insight  into  desirable  performance 
standards . 
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b.  TACCO/ Navigator.  Five  out  of  six  instructors  indicated  that  the 
number  of  WST  sessions  should  he  increased  and  th<-  number  of  ASW 
sessions  decreased,  especially  for  the  TACCO  position.  The  sixth 
instructor  did  not  feel  the  questionnaire  was  a  proper  vehicle  for 
discussion  of  this  issue.  The  instructors  recommended  that  the 
number  of  WSTs  be  increased  to  seven  or  eight  and  that  the  number 
of  ASWs  be  reduced  to  four  or  five  (given  that  a  submarine  is  nor¬ 
mally  not  available  for  ASW  training  missions). 

c.  Jezebel  Operator.  One  instructor  responded,  "It  is  a  fine  schedule" 
and  another  suggested  "Increase  aircraft  time  with  a  real  world  sub¬ 
marine.  "  The  other  4  Jezebel  instructors  all  recommended  an  in¬ 
crease  in  trainer  sessions  and  a  decrease  in  flights  (one  even  sug¬ 
gested  nine  WST  sessions  and  only  three  ASW  missions).  Localiza¬ 
tion  procedures  were  indicated  as  one  area  which  would  particularly 
benefit  from  additional  WST  training. 

d.  Radar/MAD  Operator.  The  consensus  of  opinion  was  that  these  stu¬ 
dents  derive  little  benefit  from  the  present  training  program  because 
of  the  nature  of  the  WST  and  ASW  syllabi  and  the  low  fidelity  simula¬ 
tions  of  target  characteristics.  If  the  present  training  program  and 
simulation  fidelities  are  maintained,  itwas  recommended  that  the 
participation  of  these  students  be  reduced  or  eliminated. 

OTHER  RECOMMENDATIONS.  All  comments  made  by  instructor  personnel  to 
the  final  questionnaire  item  fell  into  one  of  the  above  categories  with  one  ex¬ 
ception:  comments  regarding  the  14B3t>  device.  This  is  a  120-pound  device 
which  must  be  installed  in  the  aircraft  prior  to  each  flight  and  removed  at  flight 
conclusion.  It  is  used  by  instructor  personnel  to  control  target  information  pre¬ 
sented  to  the  students  and  is  necessitated  by  the  use  of  merchant  vessels  which 
do  not  provide  submarine  signal  characteristics.  It  was  recommended  that  the 
device  be  permanently  installed  (considerable  effort  is  required  to  remove  and 
replace  the  device)  and  that  it  be  redesigned  or  modified  to  present  higher 
fidelity  signals  to  the  students. 

TRAINING  DEVICE  COMPARISONS.  Training  effectiveness  and  device  realism 
questionnaires  have  now  been  administered  to  the  instructor  personnel  for  three 
team  training  devices:  S2E  (an  ASW  aircraft  WST),  Carrier  Air  Traffic  Con¬ 
trol  Center  (CATCC),  and  P3A/B.  Although  several  of  the  questions  in  each 
questionnaire  were  unique  to  the  individual  device  and  associated  training  pro¬ 
gram,  there  were  some  commonalities  with  respect  to  either  the  collective  in¬ 
tent  of  certain  questions  or  the  questions  per  se.  Instructor  responses  per 
device  are  presented  in  tables  23  and  24  for  those  interests  and  questions 
where  comparisons  appeared  to  be  meaningful.  It  is  suggested  that  the  dif¬ 
ferences  in  trainer  effectiveness  values  is  a  function  of  the  match  between 
simulation  fidelity  for  team  positions,  individually  and  collectively,  and  the 
training  objectives  selected  ana/or  quiet  precedence.  Information  displays 
and  other  inputs  to  a  position  have  several  characteristics,  subsets  of  which 
are  relevant  to  individual  training  objectives.  It  is  suggested  that  the  P3A/B 
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TABLE  2L  COMPARISON  OF  THE  P1A/B  AND  CATCC  TRAINER  WITH 
RESPECT  TO  AVERAGED  TASK  ENVIRONMENT  REALISM,  TASK 
PERFORMANCE  REALISM,  AND  TRAINING  EFFECTIVENESS 
CHARACTERISTICS  EVALUATIONS 


Task 

Environment 

Task 

Performance 

T  raining 

Realism 

Realism 

Effectiveness 

P3A/B 

3.1 

3.  9 

3.6 

CATCC 

4.  0 

3.9 

4.  4 

TABLE  24.  COMPARISON  OF  THE  S2E,  P3A/B  AND  CATCC  TEAM 
TRAINER  EVALUATIONS  WITH  RESPECT  TO  SPECIFIC  REALISM 
AND  EFFECTIVENESS  QUESTIONS 


Trainers 

Questions 

S2E 

P3A/B 

CATCC 

Realism: 

Information  Displays 

3.7 

2.  8 

4.  7 

Training  Effectiveness: 

Overall 

3.5 

3.  1 

4.6 

Compared  to  OJT  (Inflight,  On¬ 
board  Ship) 

3.6 

3.3 

4.  3 

Team  Coordination 

3.9 

4.  4 

4.5 
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trainer  very  adequately  simulates  those  characteristics  of  information  inputs 
to  the  TACCO  position,  relevant  to  TACCO  training  objectives,  and  to  all  posi 
tions  in  terms  of  the  team  coordination  training  objectives.  It  did  not  as  ade¬ 
quately  simulate  inputs  to  the  other  positions  in  terms  of  their  ind.vidual  train 
ing  requirements  (e.g.,  interpretation  skills).  Undoubtedly  trainer  effective¬ 
ness  depends  largely  on  the  extent  to  which  trainer  design  implements  clearly 
defined  training  requirements. 
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SECTION  V 

CONCLUSIONS  AND  RECOMMENDATIONS 


CONCLUSIONS 

TACTICS  TRAIN  TIG  OCCURS  PREDOMINANTLY  IN  THE  WST.  It  was 
apparent  from  the  study  data,  questionnaire  responses,  and  direct  observa¬ 
tion  of  training  activities  that  the  most  beneficial  ASW  tactics  training  occurs 
in  the  WST.  The  ASW  students  seldom  encounter  an  actual  submarine  during 
their  airborne  training  sessions.  Therefore,  the  simulator  is  usually  the 
only  means  of  introducing  prospective  ASW  crews  to  the  characteristics  of 
submarines  operating  in  a  submerged  environment.  The  major  drawback  to 
the  WST  is  that  it  does  not  provide  a  high-fidelity  representation  of  nuclear 
submariqe  operating  characteristics.  Nuclear  submarines  are  simulated  by 
causing  diesel  submarine  targets  to  operate  at  higher  speeds.  Nuclear  sub¬ 
marine  training  tapes  are  available,  but  the  WST  instructors  do  not  consider 
them  to  be  useful  in  the  training  situation  because  most  of  the  operating  char¬ 
acteristics  of  the  simulated  submarine  cannot  be  varied, 

TRAINING  OCCURS  AND  POSITIVE  TRANSFER  IS  EVIDENT.  Although  the 
number  of  subjects  was  small  in  most  of  the  data  analyses,  the  general  trend 
clearly  indicated  that  learning  takes  place  in  the  simulator  and  that  there  is 
positive  transfer  to  the  airborne  environment.  It  was  also  evident  that  the 
TACCO  receives  the  major  benefit  from  the  WST  training  sessions.  The 
navigator  benefits  to  a  lesser  extent,  as  does  the  Julie  operator.  The  radar / 
MAD  operator  gains  little  or  no  benefit.  The  worst  training  conditions  exist 
for  the  Jezebel  operator.  In  the  case  of  the  Jezebel  operator,  it  is  pos¬ 
sible  for  the  operator  to  experience  negative  transfer  as  he  transitions  from 
the  WST  to  the  airborne  environment  as  a  function  of  equipment  differences 
and  lack  of  JEZ  program  fidelity. 

POSSIBILITY  OF  NEGATIVE  TRANSFER  OF  THE  JEZEBEL  OPERATOR'S 
POSITION.  The  possibility  for  negative  transfer  to  occur  at  the  Jezebel 
operator's  position  existed  because  different  Jezebel  equipment  appeared  in 
the  simulator  than  in  the  aircraft.  The  simulator  incorporated  the  AQA-5 
version  of  the  equipment.  An  operator  transitioning  to  an  AQA-7  equipped 
aircraft  was  confronted  by  entirely  new  equipment  which  was  considerably 
more  sophisticated  than  the  equipment  encountered  in  the  simulator.  If  he 
was  not  scheduled  to  transition  to  an  AQA-7  equipped  aircraft,  he  often  re¬ 
ceived  his  airborne  training  in  an  aircraft  equipped  with  an  AQA-4  system. 

This  latter  equipment  was  considerably  less  sophisticated  than  the  AQA-5, 
Recent  personal  communication  with  training  squadron  instructors  indicates 
that  many  of  these  problems  will  be  eliminated  when  the  DIFAR  retrofit  equip¬ 
ment  are  incorporated  into  the  aircraft  and  the  simulator, 

LACK  OF  TRAINING  FOR  THE  ORDNANCE  MAN.  The  ordnance  man  is  gen¬ 
erally  excluded  from  the  WST  sessions  because  of  crowded  conditions  in  the 
trainers.  Inappropriate  responses  by  the  ordnance  student  can  completely 
disorganize  a  training  problem.  Consequently,  in  order  to  reduce  confusion 
in  the  trainer,  he  is  usually  invited  to  attend  only  a  single  WST  training  session. 
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OVERCROWDED  CLASS  CONDITIONS.  A  serious  training  problem  exists 
because  the  student  population  is  too  high  for  each  class.  Team  training  de¬ 
vices  are  generally  utilized  according  to  a  syllabus  and  time  schedule  designed 
to  train  one  complete  team.  The  availability  of  training  devices  and  aircraft 
and  the  flexibility  of  the  ASW  training  schedules  is  limited.  The  availability 
of  instructor  personnel  is  limited  by  the  specialized  technical  background  and 
skill  requirements  for  those  positions.  In  short,  the  P3A/B  training  program 
cannot  easily  accommodate  any  deviations  from  one  complete  crew  per  in¬ 
structor,  and  one  crew  member  per  crew  position. 

RECOMMENDATIONS 

STUDENT  FLOW  MANAGEMENT.  It  is  strongly  recommended  that  the  agen¬ 
cies  responsible  for  managing  student  flows  which  ultimately  merge  into  a 
crew  training  program  carefully  review  management  policies  and  flow  opera¬ 
tions  to  determine  how  the  student  input  requirements  for  these  programs  can 
be  rendered  more  realistic.  It  is  also  recommended  that  consideration  be 
given  to  ways  of  easing  the  training  equipment  and  schedule  constraints  within 
which  training  cadres  must  operate.  Certainly  the  provision  of  additional 
training  equipment,  support  systems  (e.g.,  software  capabilities),  and  per¬ 
sonnel  would  create  a  training  environment  better  able  to  accommodate  com¬ 
plex  ASW  training  requirements.  The  degradation  of  training  completeness 
and  effectiveness  resulting  from  student  overloads  could  lead  to  undertrained 
personnel  and  potentially  serious  operational  consequences. 

TEAM  COORDINATION  TRAINING.  It  is  recommended  that  greater  emphasis 
be  devoted  to  team  coordination  training.  It  is  recommended  that  the  syllabus 
be  revised  to  emphasize  scenarios  requiring  interaction  between  team  members. 
Student  evaluation  could  also  be  based  on  interaction  criteria.  For  example, 
performance  could  be  assessed  on  the  basis  of  such  parameters  of  crew  inter¬ 
action  as  timeliness  of  response,  provision  of  information  critical  to  problem 
solution,  method  of  presenting  critical  information  to  the  crew  members,  ob¬ 
taining  and  managing  information  sources,  etc. 

TRAINING  OF  SENSOR  OPERATORS.  The  sensor  operators  have  essentially 
three  functions:  interpreting  the  incoming  data,  entering  both  sensor  and  mis¬ 
sion  information  into  logs  and  onto  the  sensor  tapes  so  that  it  can  be  analyzed 
after  mission  completion,  and  providing  data  interpretations  in  the  form  re¬ 
quired  to  expedite  accomplishment  of  mission  objectives.  The  trainer  is  con¬ 
sidered  effective  for  training  the  latter  function.  It  is  not  considered  effective 
for  training  the  first  function,  sensor  data  interpretation.  Sensor  data  inter¬ 
pretation  is,  in  all  cases,  based  primarily  on  physical  characteristics  of  the 
signal  which  vary  as  a  function  of  target  parameter  values.  The  student  sensor 
operator  cannot  learn  to  correctly  interpret  sensor  data  unless  he  has  a  high- 
fidelity  representation  of  the  manner  in  which  these  signal  characteristics  vary. 
It  is  unlikely  that  the  presence  of  other  team  members  will  have  much  direct 
bearing  on  his  acquisition  of  interpretation  skills. 

The  P3A/B  trainer  does  not  presently  provide  high-fidelity  signals  to  the 
sensor  positions,  primarily  as  a  result  of  maintenance  and  design  limitations. 
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If  the  P3  DLFAR  retrofit  trainers  do  provide  such  signals,  it  is  recommended 
that  signal  interpretation  training  be  given  in  the  device  to  the  sensor  operator 
positions  individually  (e.g.,  one  or  two  sessions  for  only  the  Jezebel  students, 
etc,  )  with  the  instructors  for  each  position  attempting  to  make  maximum  use  of 
the  device  for  training  that  function  (e.g.,  demonstration,  presentation  of 
problems,  etc.  ).  Thereafter,  additional  training  should  occur  within  the  team 
training  context. 

MODIFICATION  OF  TACCO  TRAIN ING.  The  TACCO  does  not  deal  with  signal 
characteristics  directly,  instead  he  relies  on  interpretations  supplied  by  novice 
sensor  operators.  Thus,  the  student  TACCO  must  always  operate  under  the 
assumption  that  an  error  probability  is  associated  with  sensor  data  interpreta¬ 
tion.  For  this  reason,  it  is  recommended  that  the  TACCO  students  receive  two 
or  three  extra  sessions  wherein  the  sensor  operator  positions  are  filled  by  in¬ 
structor  personnel.  Application  of  this  procedure  would  clearly  facilitate  re¬ 
finement  of  student  TACCO  operating  techniques. 

ORDNANCE  TRAINING.  The  ordnance  man  can,  and  often  does,  degrade  actual 
ASW  operations.  The  ordnance  man  interacts  with  the  TACCO  and  to  a  small 
degree  with  the  pilot.  In  order  to  assure  that  ordnance  man  performance  degra¬ 
dations  are  reduced,  it  is  recommended  that  this  student  be  provided  with  addi¬ 
tional  WST  training  during  team  training  operations. 

INSTRUCTOR  TRAINING.  At  the  present  time,  WST  instructors  do  not  receive 
formal  instruction  in  training  device  capabilities  with  respect  to  control  of 
problem  variables  or  how  to  utilize  these  to  implement  training  objectives. 

The  extent  to  which  an  instructor  is  aware  of  device  capabilities  and  uses  is 
presently  a  function  of  relatively  informal  on-the-job  training  and  his  own 
curiosity  and  persistence.  It  is  recommended  that  a  training  course  in  device 
utilization  be  formally  structured  and  administered  with  periodic  demonstrations 
by  the  instructor  that  device  operating  and  instructing  proficiency  has  been 
maintained. 

INCREASE  TOTAL  NUMBER  OF  WST  SESSIONS.  The  training  device  presents 
a  quieter  training  environment  facilitating  student-instructor  interaction.  It 
also  provides  a  problem  "freeze"  capability  and  a  real-time  mission  plot  which 
enables  the  instructor  to  provide  the  student  with  immediate  feedback  regarding 
the  adequacy  of  his  performance.  Because  the  WST  presents  a  higher  fidelity 
situation  to  the  TACCO  and  most  of  the  other  members  of  the  crew  than  does 
the  training  aircraft,  it  is  recommended  that  the  number  of  WST  sessions  be 
increased  by  at  least  one.  It  is  also  recommended  that  the  number  of  airborne 
training  sessions  be  reduced  by  at  least  one.  Reduction  of  the  total  number  of 
training  flights  accompanied  by  an  increase  in  the  total  number  of  WST  sessions 
should  result  in  improved  ASW  tactics  training  at  less  cost  to  the  Navy. 

DESIGN  OF  FUTURE  EXPERIMENTS.  The  current  series  of  experiments  were 
on  the  threshold  of  completing  a  true  transfer  of  training  study.  It  is  recom¬ 
mended  that  in  future  similar  investigations  a  training  squadron  be  sought  that 
v/ill  "bite  the  bullet"  and  allow  one  group  of  students  to  complete  a  full  course 
of  instruction  without  access  to  the  WST.  The  airborne  ASW  performance  of 
these  crews  could  then  be  compared  with  a  like  group  having  been  trained  in 
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both  the  simulator  and  aircraft.  This  situation  was  established  for  only  the 
first  WST  session  of  the  current  study.  Although  the  data  from  these  ses¬ 
sions  were  promising,  greater  confidence  could  be  placed  in  the  obtained  re¬ 
sults  if  better  control  had  been  obtained  over  the  experimental  environment. 

In  addition  to  the  WST/non-WST  condition,  it  would  be  extremely  useful  to 
test  airborne  ASW  performance  against  actual  submarine  targets.  This  would 
allow  a  full  test  of  the  transfer  of  training  parameter  as  it  relates  to  the 
"ultimate"  performance  criterion. 
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APPENDIX  A 

REPORT  ON  2C23  SIMULATOR  EVALUATION 


EVALUATION  OF  2C23  SIMULATOR 

The  2C23  simulator,  located  in  Bldg.  453  at  the  Patuxent  Ri"er  Naval  Air 
Station,  is  a  cockpit  procedures  trainer  for  the  P-3  aircraft.  It  is  used  to 
familiarize  beginning  students  with  the  location  of  the  various  switches  in  the 
cockpit  and  the  procedures  involved  in  normal  aircraft  operations. 

The  trainer  consists  of  simulated  cockpit  panels  affixed  to  a  platform 
(approximately  10  feet  by  10  feet).  A  railing  that  is  approximately  3-1/2  feet 
high  extends  around  the  perimeter  of  the  platform.  An  instructor's  panel  con¬ 
taining  a  series  of  switches  that  control  system  operating  and  warning  lights 
located  on  the  student's  cockpit  panels  is  attached  to  the  railing.  The  in¬ 
structor  panel  can  be  operated  from  either  inside  the  platform  or  from  outside. 
If  the  instructor  stands  inside  the  platform  he  is  required  to  look  away  from 
the  cockpit  to  manipulate  his  contr  >1  panel  switches.  If  the  instructor  stands 
outside  the  platform,  he  can  look  directly  into  the  cockpit  while  operating  his 
control  panel. 

None  of  the  lights  in  the  cockpit  can  be  activated  by  the  student  as  he  manipu¬ 
lates  the  cockpit  panel  switches.  The  student's  panel  lights  can  only  be  illumi¬ 
nated  from  the  instructor's  control  panel. 

SIMULATOR  UTILIZATION  PROBLEMS 

Interviews  with  several  2C23  instructors  produced  the  following  complaints 
related  to  use  of  the  simulator  in  the  instructional  process. 

1.  INSTRUCTOR  PANEL  LOCATION.  Because  the  student's  cockpit  lights 
can  only  be  operated  from  the  instructor's  panel,  a  significant  portion  of  the 
instructor's  time  is  devoted  to  watching  the  student  manipulate  cockpit  switches. 
In  order  to  keep  the  student's  switch  manipulation  constantly  in  view,  the  in¬ 
structor  is  required  to  operate  his  simulator  control  panel  from  outside  the 
platform.  The  instructors  do  not  like  this  feature  of  the  simulator.  From  an 
instructional  point  of  view,  they  prefer  to  stand  inside  the  platform  where  stu¬ 
dent  performance  can  be  more  easily  monitored  and  where  complex  procedures 
can  be  demonstrated  in  detail.  Whenever  possible,  two  instructors  are  used 

on  the  trainer;  one  to  operate  the  instructor  panel,  the  other  to  provide  de¬ 
tailed  explanations  of  complex  procedures. 

2.  SEQUENCE  OF  SWITCH  MANIPULATION.  Several  aircraft  operating 
procedures  involve  momentary  panel  light  illumination  to  indicate  correct  com¬ 
pletion  of  a  procedural  step.  When  a  rapid  sequencing  of  switches  and  lights 

is  involved,  an  instructor  must  be  highly  current  on  simulator  operations  to 
effect  a  realistic  sequencing  of  systems  performance  indicators.  The  instruc¬ 
tors  point  out  that  it  is  fairly  difficult  to  maintain  the  desired  level  of  control 
panel  proficiency.  This  inability  to  remain  current  on  the  operation  of  the 
instructor's  panel  is  largely  a  function  of  the  time  lapse  between  classes. 
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5.  INSTRUCTOR  PANEL  SWITCHES 

a.  The  fuel  filter  indicator  light  and  the  low-pressure  indicator  (the 
latter  indicates  centrifugal  fuel  boost  pump  failure)  are  controlled 
by  the  same  switch.  The  instructors  prefer  a  single  switch  for 
each. 

b.  The  hydraulic  oil  pressure  switch  is  mislabeled. 

4.  COCKPIT  PANEL  SWITCHES 

a.  The  RPM  switches  (4)  and  oil  cooler  switches  (4)  have  a  different 
configuration  than  those  on  the  aircraft.  The  switches  on  the 
simulator  are  small,  chrome-covered  paddle  switches.  On  the 
aircraft,  they  are  large,  grey,  plastic  paddle  switches. 

b.  The  oil  cooler  switches  should  be  spring-loaded  from  "closed*1 

to  "neutral"  or  "off"  on  the  trainer.  In  the  present  mockup,  they 
do  not  operate  this  way. 

c.  On  the  overhead  panel,  the  lights  test  switch  for  the  APU  should 
be  a  pushbutton  switch. 

d.  In  the  simulator,  the  APU  inflight  arming  switch  is  upside  down 
in  relation  to  the  way  it  is  positioned  in  the  aircraft.  Since  it  is 
also  depicted  upside  down  in  the  NATOPS  manual,  the  instructors 
have  been  unable  to  effect  a  change  in  the  switch  configuration 
included  in  the  simulator. 

RECOMMENDATIONS  FOR  IMPROVING  TRAINER  UTILITY 

One  feature,  developing  functional  cockpit  switches,  was  deemed  most 
likely  to  enhance  trainer  utility.  Other  desirable  features  alluded  to  during 
these  conversations  can  be  lumped  under  the  heading  of  features  that  would  be 
"nice  to  have.  "  These  other  features  consisting  mostly  of  additional  switching 
and  circuit  breaker  capabilities  appear  below: 

a.  If  the  cockpit  switches  could  be  made  functional,  the  instructors 
would  like  to  be  able  to  activate  panel  lights  from  either  the  instruc¬ 
tor  or  cockpit  panels  or  both.  This  would  require  the  introduction 
of  a  remote/local  switch  on  the  instructor's  control  panel. 

b.  A  few  procedures  such  as  engine  restart,  shutdown,  etc.,  involve 
activating  several  switches  on  the  instructor  panel.  These  switches 
should  be  activated  simultaneously.  At  present,  they  can  only  be 
activated  in  sequence.  To  overcome  this  drawback,  the  instructors 
would  like  to  have  a  single  master  switch  installed  for  these  proce¬ 
dures  on  the  instructor  panel.  This  would  enable  a  single  switch 

to  perform  the  same  function  currently  performed  by  a  combination 
of  switches. 
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c.  A  few  procedures  involve  activating  several  switches  on  the 
instructor  panel  in  rapid  sequence.  This  creates  problems  be¬ 
cause  the  switches  are  not  grouped  together.  The  instructors 
indicated  that  relocation  of  the  switches  was  not  feasible. 

However,  they  were  inclined  to  think  color  coding  various 
switch  combinations  for  critical  procedures  might  aid  in  re¬ 
ducing  "search"  time. 

d.  The  instructors  indicated  that  inclusion  of  the  flight  essential  AC 
and  DC  circuit  breakers  on  the  forward  load  center  (panel  to  the 
rear  of  the  copilot  station)  woxild  have  a  beneficial  influence  on 
instruction. 

e.  Actual  "test"  switches  to  turn  on  all  lights  on  a  particular  panel 
were  thought  to  be  desirable. 

f.  The  following  additional  controls  were  also  indicated  as  desirable: 

*  .  Ground  or  external  power  switch 

2.  Boost  handles  on  the  bottom  of  the  center  control  stand 

3.  Cabin  pressure  control  on  the  overhead  panel 

4.  Pushbuttons  (4)  for  pressure  circuit  override  to  replace 
drawings  of  these  switches 

5.  Regulation  start  selector  and  starter  button. 

GENERAL  COMMENT 

The  bulk  of  the  interview  data  on  the  2C23  trainer  was  obtained  from  two 
instructors.  Other  instructional  staff  members  from  VP-30  c  .fered  some 
general  comments.  However,  most  of  these  centered  on  making  the  cockpit 
switches  functional.  From  the  viewpoint  of  the  two  Bunker  Ramo  observers, 
the  2C23  trainer  appears  to  be  effective  in  the  role  for  which  it  was  designed. 

It  is  apparent  from  the  interviews  that  the  instructional  staff  desire  a  number 
of  modifications  to  the  current  model  of  the  simulator.  It  is  recommended  that 
modifications  which  ar>  id  expensive  be  accomplished.  If  a  modification  re¬ 
quires  significant  simulator  redesign  or  expenditure  of  funds,  it  can  be  classed 
as  unnecessary.  Although  the  simulator,  in  its  current  configuration,  presents 
the  instructor  with  some  complex  task  requirements,  they  do  not  exceed  his 
manipulatory  capabilities. 
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APPENDIX  B 

DATA  COLLECTION  CHECKLISTS 
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DATA  COLLECTION  COORDINATOR  (DCC) 

General  Instructions 

The  DCC  will  be  responsible  for  the  overall  data  collection  effort 
among  himself,  the  simulator  operator,  and  enlisted  instructor,  during 
each  Weapons  System  Trainer  (WST)  session,  in  accordance  with  the 
Master  Schedule  contained  on  page  81. 

In  addition  to  coordinating  data  collection,  the  DCC  will  complete  the 
attached  checklist,  or  portions  thereof,  during  each  WST  session.  Check¬ 
list  entries  will  include  time  and  Pilot  Bearing  Distance  Heading  Indica¬ 
tor  readings  at  specified  points  during  the  WST  mission. 

Finally,  the  DCC  will  be  responsible  for  collecting,  arranging  and 
scoring  the  following  data  items  at  the  end  of  each  WST  session. 

(1 )  Navigator  Log 

(2)  Navigator  Dead  Reckoning  Trace  (DRT) 

(3)  Jezebel  Operator's  Log 

(4)  AQA-5  Recorder  Synchronization  Chart 

(5)  Julie/ECM  Operator's  Log 

(6)  ASA-20  Recorder  Tape 

(7)  Radar  Operator's  Log 

(8)  RO-32  Magnetic  Distortion  Recorder  Tape 

(9)  Graphic  Presentation  Overlay  Trace 

(10)  Completed  Simulator  Operator  Checklist 

(11)  Completed  Enlisted  Instructor  Checklist 

(12)  Completed  Data  Collection  Coordinator  Checklist. 
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MASTEP SCHEDULE 


WST  Session  Event  Data  Collector 


1  through  5 

Problem  Insert 

Simulator  Operator 

1  through  5 

Nav  Stab 

Crew 

PPC 

TACCO 

Nav 

Simulator  Operator 
DCC 

Enlisted  Instructor 
Enlisted  Instructor 

1,  4,  and  5 

Search 

1 ,  4 ,  and  5 

LOFAR 

Crew 

Jezebel  Operator 
PPC 

TACCO 

Nav 

Julie 

Simulator  Operator 
Simulator 

DCC 

Enlisted  Instructor 
Enlisted  Instructor 
Simulator 

Radar 

Crew 

Radar 

Simulator  Operator 

Operator 

Simulator 

TACCO 

Nav 

PPC 

Enlisted  Instructor 
Enlisted  Instructor 
DCC 

ECM  Triangulation 
Crew 

ECM 

Simulator  Operator 

Operator 

Simulator 

TACCO 

Nav 

PPC 

Enlisted  Instructor 
Enlisted  Instructor 
DCC 

2,  4,  and  5 

CODAR 

Crew 

Jezebel 

Simulator  Operator 

Operator 

TACCO 

Nav 

Julie 

Simulator 

Enlisted  Instructor 
Enlisted  Instructor 
Simulator 

PPC 

DCC 

Output 


Checklist 

Checklist 

Checklist 

Checklist 

Checklist 


Checklist 
AQA-5  Tape 
Checklist 
Checklist 
Checklist 
Log/Graphic 
Presentation 


Log/Graphic 

Presentation 

Checklist 

Checklist 

Checklist 

Checklist 

Log/Graphic 

Presentations 

Checklist 

Checklist 

Checklist 

Checklist 

AQA  -5  Tape 
Checklist 
Checklist 
Log/Graphic 
I^resentation 
Checklist 
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WST  Session 


2,  3,  and  5 


3,  4,  and  5 


3 ,  and  5 


MASTER  SCHEDULE  (continued) 


Event  Data  Collector 


Julie 

Crew 

Julie 

Simulator  Operator 

Operator 

TACCO 

Nav 

J  ezebel 

Simulator 

Enlisted  Instructor 
Enlisted  Instructor 
Simulator 

PPC 

DCC 

Active  Buoy- 
Crew 

Jezebel 

Simulator  Operator 

Operator 

Simulator 

Julie 

Operator 

TACCO 

Nav 

PPC 

Simulator 

Enlisted  Instructor 
Enlisted  Instructor 
DCC 

MAD 

Crew 

MAD 

Simulator  Operator 

Operator 

TACCO 

Nav 

PPC 

Simulator 

Enlisted  Instructor 
Enlisted  Instructor 
DCC 
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Output 


Checklist 

ASA-20  Tape 
Checklist 
Checklist 
Log/Graphic 
Presentation 
Checklist 

Checklist 

Log/Graphic 

Presentation 

ASA-20  Tape 
Checklist 
Checklist 
Checklist 

Chcckli  si 

P  O  -  $  2  Tape 
Cnecklist 
Checklist 
Checklist 
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Date: 

DCC  CHECKLIST 

Class: 

Crew: 

WST  No.  : 

Name: 

(  ) 

Check  appropriate  mission  parameters  initiated 

(  )  Target  Nos.  4,  2,  and  3  as  appropriate 
(  )  Primary  target  heading 
(  )  Primary  target  speed 

(  )  Surface/subsurface  status  of  primary  target 
(  )  ASW  aircraft  located 
(  )  Sound  characteristics  set 

(  )  Winds  Set 

(  )  Associated  sea  state  set. 

(  )  Mission  time-hack  taken  among  data  collection  team 
Remarks: 
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Date:  _ Crew:  _ WST:  _ 

EVENT:  ASW  Mission  to  include  Nav  Stab,  Search,  Localization,  and 
Attack. 

MEASURING  EQUIPMENT:  Patrol  Pilot  Commander  (PPC)  Bearing 
Distance  Heading  Indicator  (BDHI). 

DCC  ACTION:  Record  time  and  BDHI  indicators  at  "Mark  on  Top,  " 

Mark  at  Buoy  drop,  and  Weapons  Store  drop  as  follows: 

(1)  During  Nav  Stab,  record  time  and  BDHI  indicator  readings  at 
each  "Mark  on  Top." 

(2)  During  Search  (LOFAR,  Radar,  ECM),  record  time  and  BDHI 
indicator  readings  when  CODAR,  Julie,  or  Active  Buoy  marker 
buoy  is  dropped. 

(3)  During  CODAR  Localization,  record  time  and  BDHI  indicator 
readings  when  Julie  or  Active  Buoy  marker  buoy  is  dropped. 

(4)  During  Julie,  Active  Buoy  or  MAD  Pattern,  record  time  and 
BDHI  indicator  readings  when  weapons  store  is  dropped. 

Event  and  Type  BDHI 

Nav  Stab  Search  Localization  Attack  Time  A/C  Ground  Trk  Tgt  Trk  Counter 
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SIMULATOR  OPERATOR  INSTRUCTIONS 

You  are  asked  to  complete  the  attached  checklist,  or  portions 
thereof,  during  each  Weapons  System  Trainer  (WST)  session. 

The  checklist  contains  four  pages.  Each  page  representing  one, 
or  more,  phases  of  the  WST  Mission.  You  are  asked  to  complete  page  1 
at  the  beginning  of  each  session.  You  are  asked  to  complete  pages  2, 

3,  and  4,  as  applicable.  At  the  top  of  those  pages  are  noted  the  Event, 
required  Simulator  Equipment  Settings,  and  your  Data  Collection 
Procedures.  The  remainder  of  each  page  is  to  be  used  for  data  recording. 

Please  review  the  entire  checklist  and  indicate  questions  you  may 
have  concerning  it. 


86 


1 


NAVTRAEQUIPCEN  70-C-0258-2 


SIMULATOR  OPERATOR  CHECKLIST 


Date: 

Class: 

Crew: 

WST  No.: 

Name: 

ProblemStart  Time: 

Problem  Difficulty  Level 


(1) 

Primary  Target  No.  1: 

Course 

Speed 

Target  No.  2: 

Course 

Speed 

Target  No.  3: 

Course 

Speed 

(2) 

Winds:  Direction 

Speed 

Other  Sea  State  Info: 


(3)  Sound  Characteristics: 


(4)  Median  Range  of  the  Day: 
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Date:  _ Crew:  _ WST:  _ _ 

EVENT:  Nav  Stab 

MEASURING  EQUIPMENT:  Graphic  Presentation  Bearing/Distance 
Indicator 

OPERATOR  ACTION:  Select  Aircraft/Marker  Buoy  readouts  on  Graphic 
Presentation  indicator.  Record  Mark  on  Top  Number,  Time,  and 
Bearing/Distance  reading  at  each  i:Mark,  "  in  the  spaces  provided. 


If  you  are  directed  to  initiate  any  changes  in  target  parameters,  i.  e.  , 
speed,  heading,  or  surface/subsurface  status,  please  record  the  time 
and  nature  of  change(s). 


Mark  on  Top 

Parameter  Changes 

Number 

Problem  Time 

Bearing 

Distance 
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Date:  _ Crew:  _ WST:  _ _ 

EVENT:  LOFAR/RDR/ECM/ CODAR 

MEASURING  EQUIPMENT:  Graphic  Presentation  Bearing/Distance 
Indicator. 


OPERATOR  ACTION:  Select  Aircraft/Target  readouts  on  Graphic  Pre¬ 
sentation  Indicator.  Record  Bearing,  distance,  and  time  on  drop  of 
marker  "checker  buoy"  at  CODAR,  Julie,  or  Active  Buoy  pattern. 
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Date:  _ Crew: _ WST:  _ 

EVENT :  Julie/Active  Buoy/MAD 

MEASURING  EQUIPMENT:  Graphic  Presentation  Bearing/Distance 
Indicator. 


OPERATOR  ACTION:  Select  Aircraft/Target  readouts  on  Graphic  Pre 
sentation  Indicator.  Record  time,  and  bearing/distance  at  time  of  wea 
pons  store  drop. 
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Date: 

Crew: 


INSTRUCTOR  CHECKLIST 

_ Class: _ 

_ _  WST  No. : 


Name : 


Instructions 

You  are  asked  to  complete  the  attached  checklist  or  portions  thereof 
during  each  Weapons  System  Trainer  (WST)  session. 

.  ,The  chec’^lst  contains  three  pages.  Page  1  requests  data  concerning 
the  date,  training  crew,  WST  number,  and  yourself.  This  information 
should  be  completed  prior  to  the  start  of  each  session.  You  are  asked  to 
compiete  pages  2  and  3,  as  applicable.  At  the  top  of  each  page  are  noted 
the  Event,  Simulator  Equipment  involved,  and  Data  Collecting  Procedures, 
he  remainder  of  the  page  is  to  be  used  for  data  recording. 

Please  review  the  entire  checklist  and  indicate  questions  you  may 
have  concerning  it. 
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Date:  _ Crew:  _ WST:  _ 

EVENT:  Nav  Stab 

MEASURING  EQUIPMENT:  Navigator's  ID-995  wind  face  and  tactical 
coordinator's  bearing/distance  heading  indicator  BDHI. 

INSTRUCTOR  ACTION: 

(1)  Record  ID-995  wind  face  setting  (AM/LN2C  mode),  as  inserted 
by  Navigator  after  initial  airmass  computation  to  include  time 
of  setting  and  parameters.  If,  and  when  the  Navigator  '  hanges 
wind  face  settings  (AM/LN2C  mode),  record  the  time,  and 
parameters  of  each  change. 

(2)  Record  initial  Tactical  Coordinator  BDHI  setting  (ASA-16  mode), 
to  include  time  and  parameters.  Record,  as  they  occur,  changes 
in  BDHI  settings  (ASA-16  mode)  to  include  time  and  new 
parameters . 
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Date:  _ Crew:  _ WST: _ 

EVENT:  Scarch/CODAR/Julie/Active  Buoy/MAD 

MEASURING  EQUIPMENT:  Tactical  Coordinator's  Bearing/Distance 
Heading  Indicator  (BDHI). 

INSTRUCTION  ACTION:  Check  above  event  in  Programmer,  Problem 
time,  and  BDHI  indicators  (ASA-16  mode),  inserted  by  Tactical  Coordi¬ 
nator  as  a  fly  to  target,  for  the  purpose  of  dropping  a  marker  buoy  or  a 
weapons  store.  Record  time  and  BDHI  parameters  that  arc  changed  by 
the  Tactical  Coordinator  between  initial  insert  and  buoy/weapons  store 
drop.  _ _ 


|  Event  (</) 

BDHI 

LOFAR 

RDR 

ECM 

CODAR 

Active 

Buoy 

MAD 

Problem 

Time 

Ground 
T  rack 

A/C  to 
Target 

Counte  r 
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APPENDIX  C 

DATA  COLLECTION  FORMS  FOR  STUDY  NO.  2 
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Date: 


TACCO  Instructor: 


WST/ASW  No.: 


Crew: 


Experimenter's  Loc 


Simulator/Aircraft  initial  conditions 


(1)  Aircraft 


(2)  Submarine 


position 

altitude 


course 


speed 


(3)  Navy  Surface  Vessel 
position  _ 


course 


speed 

MDR 


_  position  _ 

_  course 

_  speed  _ 

_  start  depths  _ 

MDR  DEEP _ 

MDR  SNORKLE 

(4)  Merchant  Ship 

_  position _ 

_  course  _ 

_  speed  _ 

MDR 


WIND 


(6)  DATUM 
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JULIE  ECM  INSTRUCTORS 
PERFORMANCE  CHECKLIST 


O  5 


ci 

O 

H 

<! 

W  on 

H  W 

<;  ^ 


10! 


Call  Ranges  Reported  cn  First 
Change  Monitored  on  a  Sonobuoy 


NAVTRAEQ 


NAVTRAEQUIPCEN  70-C-0258-2 

JULIE  ECM  AURAL  INTERPRETATION 


a.  Classification 

BR 

TR 

Diesel 
Biologic  s 
No  Contact 
Other 

b.  RPM  ±  10% 

c.  Estimate  of  Source  Strength 

d.  Interpretation  Promptly- 
Reported  to  TACCO 


CORRECT  INCORRECT 


NAVTRAEQUIPCE 

JULIE  ECM  AURA 


a.  Classification 

BR 

TR 

Diesel 
Biologies 
No  Contact 
Other 

b.  RPM  ±  10% 

c.  Estimate  of  Source  Strength 

d.  Interpretation  Promptly- 
Reported  to  TACCO 
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JULIE  ECM  RECORDS 

DATE 

CREW 

OPERATOR  _ 

INSTRUCTOR 

Total  No. 
Correct  Entries 


Total 

Possible  Points 


ASA-20  RECORDS 


a.  ASA- 20  annotated  in  accord 

with  NATOPS _ 

b.  Magnetic  tape  annotated  in 

accord  with  NATOPS - 

c.  ASA- 20  tape  annotated  with 

buoy  channels  monitored _ 

d.  Julie  tape  annotated  with 

charge  numbers  or  time  of 
charge  every  3  minutes  when 
monitoring  active  buoys _ 

e.  DETS  (pulses)  echoes, 

bottoms  marked _ 

f.  Ranges  logged _ 

g.  Magnetic  tape  recording  of 
sonobuoy  audio  and  tactical 
ICS  channels  ranges,  doppler 
and  time 
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JULIE  ECM  RECORDS 

DATE _  CREW  _ 

OPERATOR  INSTRUCTOR 


Total  No. 
Correct  Entries 

Total 

Possible  Points 

1.  ASA-20  RECORDS 

a.  ASA- 20  annotated  in  accord 
wiHi  NATO  PS 

b.  Magnetic  tape  annotated  in 
arcnrrl  wif-b  NATOPS 

c.  ASA-20  tape  annotated  with 

huny  channels  monitored  _ 

d.  Julie  tape  annotated  with 
charge  numbers  or  time  of 
charge  every  3  minutes  when 
monitoring  active  buovs 

e.  DETS  (pulses)  echoes, 
bottoms  marked 

f.  Ranges  logged 

g.  Magnetic  tape  recording  of 
sonobuoy  audio  and  tactical 

ICS  channels  ranges,  doppler, 
and  time 
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JULIE  ECM  RECORDS 

DATE _  CREW  _ 

OPERATOR  INSTRUCTOR 


Total  No. 
Correct  Entries 

Total 

Possible  Points 

1.  ASA- 20  RECORDS 

a.  ASA- 20  annotated  in  accord 

with  NATO  PS 

b.  Magnetic  tape  annotated  in 

arrord  u/)tb  fJATnPS 

c.  ASA -20  tape  annotated  with 

hnny  rhannels  monitored 

d.  Julie  tape  annotated  with 
charge  numbers  or  time  of 
charge  every  3  minutes  when 
monitoring  artivo  buoys 

e.  DETS  (pulses)  echoes, 
bottoms  marked 

f.  Ranges  logged 

g.  Magnetic  tape  recording  of 
sonobuoy  audio  and  tactical 

ICS  channels  ranges,  doppler, 
and  time 

104 
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JULIE  ECM  RECORDS 

DATE _  CREW  _ 

OPERATOR  INSTRUCTOR 


Total  No. 
Correct  Entries 

Tota1 

Possible  Points 

,  ECM  RECORDS 

a.  Magnetic  tape  annotated  in 
arrnvH  with  NAT  OPS 

b.  Heading  information  complete 

1.  On  NAVCM/ECM  search 
lnf» 

2.  On  NAVCM/ECM  Intercept 
lop 

r.  Searrh  log  complete 

d.  Photographs  of  intercepts 

annotated 

e,  Intercept  log  complete 

1.  Signal  analysis  (Column 
?.R-hn) 

2.  Fixing  information  including 
time,  bearing,  aircraft 

position  , 

5,  Signal  pulse  slope 

f  Photograph  ,-if  intercept 

g.  Magnetic  tape  of  intercept  of 
interest  with  RENT  Report 
preceded  and  followed  by 
reference  tone  and  time 
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CREW  _  DATE _ 

OPERATOR  _ 

JEZEBEL  RECORDS 

a.  Heading  information  on  each  log 

1.  Total  possible  entries _ 

2.  Total  entries  completed _ 

b.  GRAM  annotated  in  accordance  with  NATOPS 

1.  Total  possible  GRAM  annotations  _ 

2.  Total  annotations  accompli  shed _ 

c.  Time  mark  «  on  GRAM  with  corresponding  log  entries  for  LOFAR 
and  CODAK 

1.  Total  entries  possible  _ _ 

2.  Number  entries  missed _ 

d.  Relator  frequency  and  related  bearing  logged 

1.  Total  entries  possible  _ 

2.  Total  entries  missed _ 

e.  Log  accuracy  and  completeness 

1.  Buoys  monitored,  total  entries  possible _ ,  No.  missed _ 

2.  Start  and  stop,  total  entries  possible  _ ,  No.  missed _ 

3.  CODAil  plant  designation  bearing,  total  entries  possible  _ 

No.  missed _ 

4.  Gained-lost  contact,  total  entries  possible  _ ,  No.  missed 

f.  Magnetic  tape  annotated  in  accord  with  NATOPS 

1,  Number  of  annotations  required _ 

2.  Number  of  annotations  completed _ 


1 06 
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RADAR /MAD  -  RECORDS 

DATE _ CREW _ 

OPERATOR  _ INSTRUCTOR  _ 

.Radar  Records 

a.  Heading  information 

(1)  Total  possible  entries  _  (2)  Total  number  of  entries  _ 

b.  Time  and  event  columns 

(1)  Total  possible  entries  _ (2)  Total  number  of  entries  _ 

c.  Contact  position  or  bearing  and  range  logged  for  the  following 
events:  SNK,  RE,  GN,  CL 

(1)  Total  possible  entries  _  (2)  Total  number  of  entries  _ 

d.  Events  GN  or  RG  evaluated 

(11  Total  possible  entries  _ (2)  Total  number  of  entries  _ 

e.  Events  GN  or  RG  contact  numbers  assigned 

(1)  Total  possible  entries  _  (2)  Total  number  of  entries  _ 

f.  Reference  point  for  all  position  entries  recorded  if  other  thm 
aircraft 

(1'  Total  possible  entries  _ (2)  Total  number  of  entries  _ 

g.  Actual  coverage,  if  less  than  i60  agrees,  contained  in  remarks 
and  maximum  range  of  surface  contact  at  altitude 

(1)  Total  possible  entries  _  (2)  Total  number  of  entries  _ 

(3)  Accuracy _ 
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MAD  RECORDS 


1 

No.  of 
Possible 
Annotations 

No.  of 
Actual 
Annotations 

1,  Mad  tape  annotated  in  accord 
with  NATOPS 

2.  Mad  tape  annotated  with 
operating  information  . 

a.  Pre  flight 

h,  Orientation ..... 

c.  HATS 

d.  Spurious  signals 

e.  Time  tape  drive  ON /OFF _ 

f.  Comex  and  Finex  times _ 
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P3B  TRAINER  EFFECTIVENESS  EVALUATION 


pnCTTIOM  ITMnTPU 


t->  :  i  —  :  i  -  *. 

x  nut/  uupuui 

TACCO 

Navigator 

Radar  /MAD  Operator 
Julie /ECM  Operator 
Jezebel  Operator 


As  part  of  the  evaluation  of  training  device  effectiveness  being  conducted 
for  the  U.  S.  Navy  we  would  like  you  to  answer  the  following  questions  as 
completely  and  honestly  as  you  can.  This  is  not  a  test  -  there  are  no 
right  or  wrong  answers.  Rather  what  we  are  after  is  your  carefully 
considered  responses  based  on  your  own  experience  and  competence. 

When  answering  these  questions,  please  consider  only  the  position 
checked  above  or  the  entire  team.  Please  be  as  detailed  in  your  responses 
as  you  can. 
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P3B  TRAINER  EVALUATION  QUESTIONNAIRE  * 


Overall,  how  effective  would  you  say  the  training  device  is  as  a 
trainer  for  this  position? 


i  ■■  -  . A 

Low 

Effectiveness 


i  . . ,t 

High 

Effectiveness 


How  effective  are  the  problems  and  problem  characteristics  in 
providing  training  for  this  position? 

i— — — - -JL - X... . . .  -  I _ I 

Low  High 

Effectiveness  Effectiveness 


Would  you  change  the  problems  or  the  problem  characteristics  in 
any  way  for  training  purposes?  If  so,  what  changes  would  you 
make  ? 


NAVTRAEQUIPCEN  70-C-0258-2. 


3.  How  effective  do  you  think  the  trainer  is  compared  to  an  equivalent 
amount  of  inflight  training? 

«• . — - 1 - - * . . 1 _ _ i 

Low  High 

Effectiveness  Effectiveness 

Are  there  any  aspects  of  the  operational  ASW  mission  (e.  g.  ,  coordina¬ 
tion,  target  information,  etc.)  for  which  the  trainer  provides  better 
training  than  the  inflight  training  missions  do?  If  so,  please  discuss 
them. 


Are  there  any  aspects  of  the  operational  ASW  mission  for  v.hich  the 
inflight  training  missions  provide  better  training  than  the  trainer 
does?  If  so,  please  discuss  them. 
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4.  How  realistic  are  the  displays  of  sensor  data  and/or  other  inform 
tion  at  this  position  in  the  trainer  ? 


t - - - 

Low 

Realism 


High 

Realism 


What  improvemonts  would  you  suggest  where? 
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5.  For  a  number  of  reasons  (e.  g.  ,  poor  or  failed  equipments,  type  of 
target  utilized,  level  of  performance  in  the  other  positions)  both  the 
WST  and  ASW  training  sessions  can  be  considered  to  be  simulations 
of  the  "real"  operational  mission.  In  question  5a  we  -would  like  you 
to  evaluate  the  realism  of  both  training  environments,  while  in 
question  5b  we  would  like  you  to  evaluate  the  effectiveness  of  both 
training  environments  for  different  stages  or  types  of  training. 

5a.  How  realistic  are  the  feedbacks  provided  by  each  of  the  training 

environments  (trainer  vs.  inflight  aircraft)  in  response  to  the  student's 
performance  inputs  during  the  problem? 

TRAINER:  AIRCRAFT: 

l,— A - . >  ..  .  ■■  >  L- . -  ..  -A, ..-..-a. . . J— _ | 

Low  High  Low  High 

Realism  Realism  Realism  Realism 


5b.  How  effective  is  each  of  the  training  environments  for  each  of  four 
types  of  training:  (1)  Demonstration  and  initial  practice,  (2)  Develop 
ment  of  proficiency,  (3)  Refinement  of  technique  once  proficiency  has 
been  acquired,  and  (4)  Development  and  practice  of  new  tactics.  (If 
you  feel  the  answer  to  this  question  depends  on  whether  one  considers 
jus t  the  position  you  are  evaluating  or  whether  one  considers,  instead, 
the  performance  of  the  entire  team  as  a  coordinated  unit,  then  please 
evaluate  both.  Use  an  "x"  for  the  position  under  consideration  and  an 
"o"  for  the  team  as  a  whole.  ) 


TRAINER 

DEMO  b  PRACT.  i  . 

1  ! 

AIRCRAFT 

I  ! 

I 

l  i 

DEV  OE  PROF  i  i  i 

i  i 

1  1 

1  _  1 

TEOHMIQIIF  I  I  1 

i  i 

1  1 

1  1 

JvJITW  TACTICS  1  1  1 

i  i 

1  1 

I 

.  1  1 

Low 

Effective  - 

ness 

High 

Effective¬ 

ness 

Low 

Effective  - 
ness 

High 

Effective¬ 

ness 
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Please  compare  and  rate  the  realism  of  the  trainer  to  the  inflight 
situation  for  the  following  eight  items.  (Notes:  (1)  If  the  use  of  a  sub 
vs.  a  surface  vessel  makes  a  difference,  then  use  an  "x"  for  the 
comparison  of  the  trainer  to  a  flight  using  a  sub  and  an  "o"  for  a 
surface  vessel  flight.  (2)  If  an  item  is  not  applicable  to  the  position 
you  are  evaluating,  please  mark  it  NA  and  proceed  to  the  next  item.  ) 

TARGET  INFORMATION  RECEIVED 

ti-.i.i- t.  i  ■  . —t i - ...» -,,1 

Low  Realism  High  Realism 

BUOY  INFORMATION  RECEIVED 

.  1 .  •  t .  » 

Low  Realism  High  Realism 

MATCH  OF  THE  ELECTRONIC  TO  THE  REAL  WORLD 

L - - - - - - -  .  1  ...  ■  * 

Low  Realism  High  Realism 

TACCO/SENSOR  OPERATOR  INTERACTION 

l~ .  L  ..  .  k  1 

Low  Realism  High  Realism 

TACCO /PILOT  INTERACTION  TO  CONTROL  GUIDANCE 

■  .......  t  . ..  X  ....  | 

Low  Realism  High  Realism 

TACCO /NAVIGATOR  ESTIMATION  OF  COURSE 

1 . .  ...  i  i  i 

Low  Realism  High  Realism 

TACCO /NAVIGATOR  ESTIMATION  OF  SPEED 

4- .  .......  i  .  t  i 

Low  Realism  High  Realism 

TACCO /NAVIGATOR  ESTIMATION  OF  POSITION  RELATIVE  TO  BUOYS 


Low  Realism 


High  Realism 
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Please  compare  and  rate  the  effectiveness  of  the  trainer  for  instruc¬ 
tion  of  a  student  in  the  following  (if  an  item  is  not  pertinent  to  the 
position  you  are  evaluating,  please  mark  it  NA  and  proceed): 


KNOBOLOGY 


Low  Effectiveness 


High  Effectiveness 


PROCEDURAL  USE  OF  EQUIPMENT 


Low  Effectiveness 


High  Effectiveness 


INTERPRETATION  OF  SENSOR  DATA 


Low  Effectiveness 


High  Effectiveness 


UTILIZATION  OF  SENSOR  INFORMATION 


Low  Effectiveness 


High  Effectiveness 


METHODOLOGY  AND  TECHNIQUE  DEVELOPMENT  («  .  g.  ,  calcula¬ 
tions,  timing,  identification  and  selection  of  alternatives) 


Low  Effectiveness 


High  Effectiveness 


TACTICS 


Low  Effectiveness 


High  Effectiveness 


TEAM  COORDINATION 


Low  Effectiveness 


High  Effectiveness 
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8.  In  terms  of  working  with  the  trainee  during  a  training  session  which 
aspects  of  the  trainer's  design  do  you  feel  aid  or  hinder  your  efforts 
as  an  instructor? 


9.  Are  there  any  modifications  or  improvements  you  would  suggest  that 
would  allow  better  or  more  efficient  evaluation  of  student  performance 
at  this  station  (i.  e.  ,  either  in  the  form  of  recording  devices  ,  better 
or  more  sla ve  equipment,  etc.)? 
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10.  What  changes  in  the  design  or  operation  of  the  device  do  you  think 
should  be  made  in  order  to  enhance  the  training  at  this  position? 


Are  there  any  changes  in  maintenance  you  would  suggest? 


11.  Would  you  change  the  amount  of  training  time  presently  spent  m  the 
trainer  or  in  the  aircraft,  or  would  you  suggest  any  different  utili¬ 
zation  of  the  two  training  environments  (e.g.  ,  scheduling)?  If  so, 
please  discuss  the  changes  you  would  make  and  explain  the  expected 
advantages. 
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12.  What  characteristics  of  the  training  device,  or  utilization  thereof ,  would 
you  say  contribute  to  its  effectiveness  or  ineffectiveness  for  train¬ 
ing  purposes  (but  have  not  been  discussed  above)? 

Effective  Characteristics  I  Ineffective  Characteristics 
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13  ABSTRACT 


Tv/o  separate  experiments  were  performed  to  evaluate  the  effectiveness 
of  the  Navy  2F69B  Weapon  System  Trainer.  These  experiments  were 
conducted  at  the  Patuxent  River  Naval  Air  Station,  Patuxent,  Maryland. 
Training  Squadron  VP-30  provided  the  WST,  aircraft,  students,  and  in¬ 
structor  personnel  used  in  both  studies.  During  the  first  study,  experi¬ 
mental  data  were  collected  on  two  classes  of  students  composed  of  six 
and  seven  crews.  During  the  second  study,  data  were  collected  on  two 
classes  of  students  composed  of  four  and  seven  crews.  The  first  series 
of  experiments  were  conducted  in  the  WST.  The  second  series  were  con 
ducted  in  both  the  WST  and  airborne  environments.  It  was  apparent 
from  the  experimental  data,  questionnaire  responses,  and  direct  obser¬ 
vation  of  training  activities  that  the  most  beneficial  ASW  tactics  training 
occurs  in  the  WST.  There  were  also  clear  indications  that  the  training 
received  in  the  WST  was  transferred  in  a  positive  manner  to  the  air¬ 
borne  environment.  As  a  result  of  these  studies,  it  is  recommended 
that  the  number  of  WST  training  sessions  be  increased  and  that  the  num¬ 
ber  of  P3  training  flights  be  reduced. 


DD  .'““.1473 


Unclassified 

Security  Classification 


Unclassified 

Security  CUftaificatlon 


K  r  Y  WORD! 


Training  Effectiveness 
ASW  Tactics  Training 
2F69B  Weapon  System  Trainer 
Airborne  Environments 


first  series  of  experiments 


